2023-01-27T12:26:45.316 INFO:root:teuthology version: 0.0.1.dev33+gbf44874 2023-01-27T12:26:45.322 INFO:teuthology.results:subset: '111/120000' 2023-01-27T12:26:45.322 INFO:teuthology.results:seed: '9535' 2023-01-27T16:13:01.336 INFO:root:teuthology version: 0.0.1.dev33+gbf44874 2023-01-27T16:13:01.388 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-27T18:29:14.159 INFO:teuthology.results:Tests finished! gathering results... 2023-01-27T18:29:38.666 INFO:scrape:Found 317 jobs 2023-01-27T18:29:38.790 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140019/teuthology.log 2023-01-27T18:31:02.450 INFO:scrape:Found 26 distinct failure reasons 2023-01-27T18:31:02.452 INFO:scrape:Failure: "2023-01-27T13:22:48.659921+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log 2023-01-27T18:31:02.452 INFO:scrape:97 jobs 2023-01-27T18:31:02.453 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.454 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-inject-leak/mon', '2-inject-leak/osd', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_latest}', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'fast/fast', 'fast/normal', 'mgr_ttl_cache/disable', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/21', 'rados/multimon/{clusters/6', 'rados/objectstore/{backends/filestore-idempotent', 'rados/objectstore/{backends/filestore-idempotent-aio-journal', 'rados/objectstore/{backends/objectcacher-stress', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/ec-inconsistent-hinfo', 'rados/singleton/{all/ec-lost-unfound', 'rados/singleton/{all/osd-recovery', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/singleton/{all/rebuild-mondb', 'rados/singleton/{all/recovery-preemption', 'rados/singleton/{all/thrash-rados/{thrash-rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-comp-zlib}', 'random-objectstore$/{filestore-xfs}', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/crash}', 'tasks/failover}', 'tasks/insights}', 'tasks/libcephsqlite}', 'tasks/mon_clock_with_skews}', 'tasks/mon_recovery', 'tasks/mon_recovery}', 'tasks/progress}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'tasks/rados_cls_all', 'tasks/rados_striper}', 'tasks/rados_workunit_loadgen_big}', 'tasks/rados_workunit_loadgen_mix}', 'tasks/rados_workunit_loadgen_mostlyread}', 'tasks/repair_test}', 'tasks/scrub_test}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/force-sync-many', 'thrashers/many', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashers/sync', 'thrashers/sync-many', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/dedup-io-mixed}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-radosbench}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/rados_mon_osdmap_prune}', 'workloads/rados_mon_workunits}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] 2023-01-27T18:31:02.454 INFO:scrape: 2023-01-27T18:31:02.455 INFO:scrape:Dead: None 2023-01-27T18:31:02.455 INFO:scrape:3 jobs: ['7140019', '7139879', '7139884'] 2023-01-27T18:31:02.456 INFO:scrape: 2023-01-27T18:31:02.456 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-01-27T18:31:02.456 INFO:scrape:32 jobs 2023-01-27T18:31:02.457 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.457 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '2-services/rgw', '3-final}', 'agent/off', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/osds/{0-distro/ubuntu_20.04', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8}', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04}', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/cephadm/smoke/{0-distro/ubuntu_20.04', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'start}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.458 INFO:scrape: 2023-01-27T18:31:02.458 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi016.front.sepia.ceph.com: ['type=AVC msg=audit(1674832059.486:15024): avc: denied { map } for pid=61354 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:419): avc: denied { open } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:420): avc: denied { lock } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.309:17275): avc: denied { read } for pid=899 comm="sssd" name="resolv.conf" dev="sda1" ino=393262 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15022): avc: denied { read } for pid=61354 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:419): avc: denied { read } for pid=7013 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15022): avc: denied { open } for pid=61354 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17379): avc: denied { map } for pid=70126 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=393227 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17377): avc: denied { read } for pid=70126 comm="rpm" name="Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17308): avc: denied { map } for pid=69887 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=393227 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:421): avc: denied { map } for pid=7013 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17377): avc: denied { open } for pid=70126 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17306): avc: denied { open } for pid=69887 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17307): avc: denied { lock } for pid=69887 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15023): avc: denied { lock } for pid=61354 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17306): avc: denied { read } for pid=69887 comm="rpm" name="Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17378): avc: denied { lock } for pid=70126 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.133:15001): avc: denied { search } for pid=6923 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] 2023-01-27T18:31:02.459 INFO:scrape:22 jobs: ['7139871', '7139722', '7139968', '7139836', '7139955', '7140016', '7139922', '7139707', '7139802', '7139891', '7139923', '7139742', '7139765', '7139731', '7139935', '7140018', '7139777', '7139990', '7139872', '7139867', '7139708', '7139795'] 2023-01-27T18:31:02.459 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.460 INFO:scrape:suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/pacific', 'agent/off', 'agent/on', 'backoff/peering', 'ceph', 'clusters/3-mons', 'clusters/{openstack', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'rados', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'rados/monthrash/{ceph', 'rados/objectstore/{backends/filejournal', 'rados/objectstore/{backends/keyvaluedb', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/singleton-nomsgr/{all/crushdiff', 'rados/singleton-nomsgr/{all/export-after-evict', 'rados/singleton-nomsgr/{all/msgr', 'rados/singleton/{all/dump-stuck', 'rados/singleton/{all/max-pg-per-osd.from-mon', 'rados/singleton/{all/mon-config-key-caps', 'rados/singleton/{all/test-noautoscale-flag', 'rados/standalone/{supported-random-distro$/{centos_8}', 'rados/standalone/{supported-random-distro$/{rhel_8}', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'task/test_adoption}', 'task/test_nfs}', 'task/test_orch_cli}', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/mgr}', 'workloads/misc}', 'workloads/osd-backfill}', 'workloads/rados_5925}', 'workloads/scrub}', 'workloads/test_rbd_api}'] 2023-01-27T18:31:02.460 INFO:scrape: 2023-01-27T18:31:02.460 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi073.front.sepia.ceph.com: ['type=AVC msg=audit(1674835730.890:216): avc: denied { map } for pid=6357 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14041): avc: denied { lock } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835983.409:14037): avc: denied { search } for pid=6244 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835730.859:214): avc: denied { read } for pid=6357 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835730.875:215): avc: denied { lock } for pid=6357 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835730.859:214): avc: denied { open } for pid=6357 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14040): avc: denied { open } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14042): avc: denied { map } for pid=77403 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14040): avc: denied { read } for pid=77403 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] 2023-01-27T18:31:02.461 INFO:scrape:22 jobs: ['7139984', '7139789', '7139725', '7139899', '7139815', '7139728', '7139961', '7139782', '7139763', '7140015', '7139818', '7139970', '7139947', '7139733', '7139946', '7139715', '7139827', '7139995', '7139746', '7139994', '7139943', '7139942'] 2023-01-27T18:31:02.461 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.462 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{openstack', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/3', 'rados/objectstore/{backends/objectstore-bluestore-a', 'rados/rest/{mgr-restful', 'rados/singleton-nomsgr/{all/balancer', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/singleton-nomsgr/{all/pool-access', 'rados/singleton/{all/deduptool', 'rados/singleton/{all/divergent_priors2', 'rados/singleton/{all/max-pg-per-osd.from-replica', 'rados/singleton/{all/mon-auth-caps', 'rados/standalone/{supported-random-distro$/{centos_8}', 'rados/standalone/{supported-random-distro$/{rhel_8}', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'task/test_orch_cli_mon}', 'tasks/mon_clock_with_skews}', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/sync', 'thrashosds-health', 'three-plus-one}', 'workloads/c2c}', 'workloads/dedup-io-snaps}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/mon}', 'workloads/rados_5925}', 'workloads/radosbench}', 'workloads/rbd_cls}'] 2023-01-27T18:31:02.462 INFO:scrape: 2023-01-27T18:31:02.463 INFO:scrape:Failure: {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 12, 'changed': True, 'cmd': ['subscription-manager', 'register', '--activationkey=testnode', '--org=Ceph', '--name=smithi088', '--force'], 'delta': '0:00:00.774678', 'end': '2023-01-27 15:41:48.141999', 'failed_when_result': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager register --activationkey=testnode --org=Ceph --name=smithi088 --force', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 70, 'start': '2023-01-27 15:41:47.367321', 'stderr': 'HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "katello_subscription_facets_host_id"\nDETAIL: Key (host_id)=(407) already exists.', 'stderr_lines': ['HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "katello_subscription_facets_host_id"', 'DETAIL: Key (host_id)=(407) already exists.'], 'stdout': '', 'stdout_lines': []}} 2023-01-27T18:31:02.463 INFO:scrape:2 jobs: ['7139924', '7139933'] 2023-01-27T18:31:02.464 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.464 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/valgrind-leaks/{1-start', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2023-01-27T18:31:02.464 INFO:scrape: 2023-01-27T18:31:02.465 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2023-01-27T18:31:02.465 INFO:scrape:2 jobs: ['7139806', '7139732'] 2023-01-27T18:31:02.466 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] 2023-01-27T18:31:02.466 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] 2023-01-27T18:31:02.466 INFO:scrape: 2023-01-27T18:31:02.467 INFO:scrape:Failure: Test failure: test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) 2023-01-27T18:31:02.467 INFO:scrape:1 jobs: ['7139972'] 2023-01-27T18:31:02.468 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zlib}', 'tasks/dashboard}'] 2023-01-27T18:31:02.468 INFO:scrape: 2023-01-27T18:31:02.468 INFO:scrape:Failure: Command failed (workunit test rados/test_librados_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh' 2023-01-27T18:31:02.469 INFO:scrape:2 jobs: ['7139754', '7139912'] 2023-01-27T18:31:02.469 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/librados_hello_world'] 2023-01-27T18:31:02.470 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/librados_hello_world', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2023-01-27T18:31:02.470 INFO:scrape: 2023-01-27T18:31:02.470 INFO:scrape:Failure: Test failure: test_token_ttl (tasks.mgr.dashboard.test_auth.AuthTest) 2023-01-27T18:31:02.471 INFO:scrape:1 jobs: ['7139812'] 2023-01-27T18:31:02.471 INFO:scrape:suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/dashboard}'] 2023-01-27T18:31:02.471 INFO:scrape: 2023-01-27T18:31:02.472 INFO:scrape:Failure: Command failed on smithi026 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' 2023-01-27T18:31:02.472 INFO:scrape:6 jobs: ['7139816', '7139854', '7139973', '7139735', '7139895', '7139856'] 2023-01-27T18:31:02.473 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.473 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '1-install/pacific', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'backoff/peering_and_degraded', 'ceph', 'cluster/1-node', 'cluster/3-node', 'clusters/{fixed-4', 'clusters/{openstack', 'd-balancer/crush-compat', 'k8s/1.21', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'net/calico', 'net/flannel', 'net/host', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'recovery-overrides/{more-async-partial-recovery}', 'rook/1.7.2}', 'rook/master}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-01-27T18:31:02.473 INFO:scrape: 2023-01-27T18:31:02.474 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1674833130.822:14037): avc: denied { read } for pid=77365 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833130.822:14037): avc: denied { open } for pid=77365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.773:213): avc: denied { map } for pid=5861 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.746:211): avc: denied { read } for pid=5861 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.770:212): avc: denied { lock } for pid=5861 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833130.823:14039): avc: denied { map } for pid=77365 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833129.565:14034): avc: denied { search } for pid=4750 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393279 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674833130.823:14038): avc: denied { lock } for pid=77365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.746:211): avc: denied { open } for pid=5861 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] 2023-01-27T18:31:02.474 INFO:scrape:11 jobs: ['7139894', '7139861', '7139911', '7139966', '7139960', '7139791', '7139736', '7139761', '7140006', '7139734', '7139814'] 2023-01-27T18:31:02.475 INFO:scrape:suites intersection: [] 2023-01-27T18:31:02.475 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/octopus', 'agent/on', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/dashboard/{0-single-container-host', 'rados/objectstore/{backends/alloc-hint', 'rados/objectstore/{backends/fusestore', 'rados/singleton-nomsgr/{all/crushdiff', 'rados/singleton-nomsgr/{all/large-omap-object-warnings', 'rados/singleton-nomsgr/{all/version-number-sanity', 'rados/singleton/{all/peer', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-comp-lz4}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'task/test_nfs}', 'task/test_orch_cli_mon}', 'tasks/e2e}', 'thrashers/none', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2023-01-27T18:31:02.476 INFO:scrape: 2023-01-27T18:31:02.476 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-01-27T18:31:02.476 INFO:scrape:2 jobs: ['7139975', '7139819'] 2023-01-27T18:31:02.477 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-01-27T18:31:02.477 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-01-27T18:31:02.478 INFO:scrape: 2023-01-27T18:31:02.478 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi175.front.sepia.ceph.com: ['type=AVC msg=audit(1674828821.236:419): avc: denied { open } for pid=6993 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15024): avc: denied { map } for pid=61334 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15023): avc: denied { lock } for pid=61334 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828821.236:420): avc: denied { lock } for pid=6993 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15022): avc: denied { open } for pid=61334 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.124:15001): avc: denied { search } for pid=6903 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674828821.236:421): avc: denied { map } for pid=6993 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15022): avc: denied { read } for pid=61334 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828821.236:419): avc: denied { read } for pid=6993 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] 2023-01-27T18:31:02.478 INFO:scrape:3 jobs: ['7139803', '7140001', '7139887'] 2023-01-27T18:31:02.479 INFO:scrape:suites intersection: ['mon_election/classic', 'rados', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.479 INFO:scrape:suites union: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/singleton/{all/pg-autoscaler', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.480 INFO:scrape: 2023-01-27T18:31:02.480 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2023-01-27T18:31:02.480 INFO:scrape:2 jobs: ['7139847', '7139710'] 2023-01-27T18:31:02.481 INFO:scrape:suites intersection: ['agent/off', 'mon_election/classic', 'task/test_cephadm_repos}'] 2023-01-27T18:31:02.481 INFO:scrape:suites union: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'task/test_cephadm_repos}'] 2023-01-27T18:31:02.482 INFO:scrape: 2023-01-27T18:31:02.482 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi133.front.sepia.ceph.com: ['type=AVC msg=audit(1674836133.166:17318): avc: denied { read } for pid=69968 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835970.084:13743): avc: denied { read } for pid=56397 comm="rhsmcertd-worke" name="python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.418:15013): avc: denied { search } for pid=6916 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835969.461:13728): avc: denied { read write } for pid=56397 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15034): avc: denied { open } for pid=61439 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17320): avc: denied { map } for pid=69968 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15034): avc: denied { read } for pid=61439 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13766): avc: denied { open } for pid=56758 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13733): avc: denied { lock } for pid=56569 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13766): avc: denied { read } for pid=56758 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13768): avc: denied { map } for pid=56758 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13767): avc: denied { lock } for pid=56758 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15035): avc: denied { lock } for pid=61439 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836132.810:17287): avc: denied { read } for pid=886 comm="sssd" name="resolv.conf" dev="sda1" ino=394924 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13730): avc: denied { getattr } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=77502 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13732): avc: denied { read } for pid=56569 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13734): avc: denied { map } for pid=56569 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17318): avc: denied { open } for pid=69968 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13731): avc: denied { map } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=77502 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13729): avc: denied { lock } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835970.084:13743): avc: denied { open } for pid=56397 comm="rhsmcertd-worke" path="/etc/dnf/modules.d/python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13732): avc: denied { open } for pid=56569 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17319): avc: denied { lock } for pid=69968 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13728): avc: denied { open } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15036): avc: denied { map } for pid=61439 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] 2023-01-27T18:31:02.483 INFO:scrape:2 jobs: ['7139996', '7139876'] 2023-01-27T18:31:02.483 INFO:scrape:suites intersection: ['mon_election/connectivity', 'rados', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.483 INFO:scrape:suites union: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton-nomsgr/{all/full-tiering', 'rados/singleton/{all/erasure-code-nonregression', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.484 INFO:scrape: 2023-01-27T18:31:02.484 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi053.front.sepia.ceph.com: ['type=AVC msg=audit(1674833013.112:15003): avc: denied { search } for pid=6903 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262150 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674833013.469:15024): avc: denied { read } for pid=61327 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15025): avc: denied { lock } for pid=61327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15026): avc: denied { map } for pid=61327 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15024): avc: denied { open } for pid=61327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] 2023-01-27T18:31:02.485 INFO:scrape:1 jobs: ['7139896'] 2023-01-27T18:31:02.485 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/large-omap-object-warnings', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.485 INFO:scrape: 2023-01-27T18:31:02.486 INFO:scrape:Failure: {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi088.front.sepia.ceph.com,172.21.15.88' (ECDSA) to the list of known hosts.\r\nubuntu@smithi088.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive)."}} 2023-01-27T18:31:02.486 INFO:scrape:1 jobs: ['7139889'] 2023-01-27T18:31:02.487 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/thrash-backfill-full', 'supported-random-distro$/{centos_8}}'] 2023-01-27T18:31:02.487 INFO:scrape: 2023-01-27T18:31:02.487 INFO:scrape:Crash: Command failed (workunit test rados/test_crash.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh' 2023-01-27T18:31:02.488 INFO:scrape:ceph version 17.2.5-691-gb0c8245b (b0c8245b45314ecfe0b7cf39532aecb29f18f697) quincy (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf9d917b20] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7faf9cf5b8f0] 4: (AsyncMessenger::wait()+0x147) [0x55f7cfda41c7] 5: main() 6: __libc_start_main() 7: _start() 2023-01-27T18:31:02.488 INFO:scrape:1 jobs: ['7139864'] 2023-01-27T18:31:02.489 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{rhel_8}}'] 2023-01-27T18:31:02.489 INFO:scrape: 2023-01-27T18:31:02.489 INFO:scrape:Failure: Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh' 2023-01-27T18:31:02.490 INFO:scrape:1 jobs: ['7139888'] 2023-01-27T18:31:02.490 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/erasure-code}'] 2023-01-27T18:31:02.490 INFO:scrape: 2023-01-27T18:31:02.491 INFO:scrape:Failure: {'smithi181.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'only_upgrade': False, 'package': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'policy_rc_d': None, 'purge': False, 'state': 'absent', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': "'apt-get remove 'librbd1' 'librados2'' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n", 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], 'stdout': '', 'stdout_lines': []}} 2023-01-27T18:31:02.491 INFO:scrape:1 jobs: ['7139992'] 2023-01-27T18:31:02.492 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}'] 2023-01-27T18:31:02.492 INFO:scrape: 2023-01-27T18:31:02.492 INFO:scrape:Failure: Test failure: test_file_sd_command (tasks.mgr.test_prometheus.TestPrometheus) 2023-01-27T18:31:02.493 INFO:scrape:1 jobs: ['7139988'] 2023-01-27T18:31:02.493 INFO:scrape:suites: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{centos_8}', 'tasks/prometheus}'] 2023-01-27T18:31:02.494 INFO:scrape: 2023-01-27T18:31:02.494 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2023-01-27T18:31:02.494 INFO:scrape:2 jobs: ['7139822', '7139953'] 2023-01-27T18:31:02.495 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_cephadm}'] 2023-01-27T18:31:02.495 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'task/test_cephadm}'] 2023-01-27T18:31:02.496 INFO:scrape: 2023-01-27T18:31:02.497 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) 2023-01-27T18:31:02.497 INFO:scrape:1 jobs: ['7139868'] 2023-01-27T18:31:02.498 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_nfs}'] 2023-01-27T18:31:02.498 INFO:scrape: 2023-01-27T18:31:02.498 INFO:scrape:Failure: Cannot connect to remote host smithi097 2023-01-27T18:31:02.499 INFO:scrape:1 jobs: ['7139878'] 2023-01-27T18:31:02.499 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{centos_8}}'] 2023-01-27T18:31:02.500 INFO:scrape: 2023-01-27T18:31:02.500 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.88 2023-01-27T18:31:02.500 INFO:scrape:1 jobs: ['7139880'] 2023-01-27T18:31:02.501 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/one', 'workloads/snaps-few-objects}'] 2023-01-27T18:31:02.501 INFO:scrape: 2023-01-27T18:31:02.513 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/ failed: 216 dead: 5 running: 0 waiting: 0 queued: 0 passed: 96 Fail ================================================================= [7139707] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139707/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139707/ SELinux denials found on ubuntu@smithi085.front.sepia.ceph.com: ['type=AVC msg=audit(1674822698.709:420): avc: denied { map } for pid=7014 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822698.709:418): avc: denied { read } for pid=7014 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822698.709:418): avc: denied { open } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822961.181:14998): avc: denied { search } for pid=6924 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674822961.527:15021): avc: denied { open } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822961.527:15021): avc: denied { read } for pid=61356 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822961.527:15022): avc: denied { lock } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822698.709:419): avc: denied { lock } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822961.527:15023): avc: denied { map } for pid=61356 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139708] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd-backfill} ----------------------------------------------------------------- time: 03:24:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139708/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139708/ SELinux denials found on ubuntu@smithi077.front.sepia.ceph.com: ['type=AVC msg=audit(1674823166.673:14036): avc: denied { search } for pid=6248 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674822920.837:215): avc: denied { open } for pid=6361 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823291.616:16966): avc: denied { read } for pid=692 comm="sssd" name="resolv.conf" dev="sda1" ino=525255 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823291.977:16999): avc: denied { map } for pid=94014 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822920.855:217): avc: denied { map } for pid=6361 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823167.980:14041): avc: denied { map } for pid=77403 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826805.652:18198): avc: denied { lock } for pid=317047 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823168.747:14061): avc: denied { open } for pid=77483 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822920.837:215): avc: denied { read } for pid=6361 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823168.748:14063): avc: denied { map } for pid=77483 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823291.977:16998): avc: denied { lock } for pid=94014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826802.283:18196): avc: denied { search } for pid=6248 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674826805.652:18197): avc: denied { read } for pid=317047 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826805.652:18197): avc: denied { open } for pid=317047 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826805.652:18199): avc: denied { map } for pid=317047 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823168.747:14061): avc: denied { read } for pid=77483 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823167.980:14039): avc: denied { read } for pid=77403 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674822920.851:216): avc: denied { lock } for pid=6361 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823168.748:14062): avc: denied { lock } for pid=77483 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823167.980:14040): avc: denied { lock } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823291.977:16997): avc: denied { read } for pid=94014 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823291.977:16997): avc: denied { open } for pid=94014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823167.980:14039): avc: denied { open } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139709] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:21:17 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139709/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139709/ "2023-01-27T12:54:17.012248+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139710] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139710/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139710/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b2b5f23ab2b421b8c6f944999bb7218 Command failed (workunit test cephadm/test_repos.sh) on smithi165 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7139712] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:25:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139712/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139712/ "2023-01-27T12:47:54.611250+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139714] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/repair_test} ----------------------------------------------------------------- time: 00:18:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139714/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139714/ "2023-01-27T12:45:20.696539+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139715] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:23:10 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139715/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139715/ SELinux denials found on ubuntu@smithi064.front.sepia.ceph.com: ['type=AVC msg=audit(1674823693.903:16992): avc: denied { lock } for pid=94022 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823693.903:16991): avc: denied { open } for pid=94022 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823693.537:16960): avc: denied { read } for pid=691 comm="sssd" name="resolv.conf" dev="sda1" ino=525302 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823569.600:14038): avc: denied { read } for pid=77410 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823569.600:14038): avc: denied { open } for pid=77410 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823316.486:215): avc: denied { lock } for pid=6368 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823316.486:216): avc: denied { map } for pid=6368 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823316.471:214): avc: denied { open } for pid=6368 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823693.903:16991): avc: denied { read } for pid=94022 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823316.471:214): avc: denied { read } for pid=6368 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823568.468:14035): avc: denied { search } for pid=6255 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674823693.903:16993): avc: denied { map } for pid=94022 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823569.600:14039): avc: denied { lock } for pid=77410 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823569.600:14040): avc: denied { map } for pid=77410 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139717] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139717/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139717/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2f8b1e50e3a404fadca9af3c5feaecd timeout expired in wait_until_healthy [7139722] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_5925} ----------------------------------------------------------------- time: 00:12:47 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139722/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139722/ SELinux denials found on ubuntu@smithi007.front.sepia.ceph.com: ['type=AVC msg=audit(1674823804.848:15056): avc: denied { read } for pid=61475 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.848:15057): avc: denied { lock } for pid=61475 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823533.761:404): avc: denied { open } for pid=7009 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823533.761:404): avc: denied { read } for pid=7009 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.119:15001): avc: denied { read } for pid=61303 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.848:15058): avc: denied { map } for pid=61475 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823803.708:14966): avc: denied { search } for pid=6936 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674823533.761:406): avc: denied { map } for pid=7009 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.119:15001): avc: denied { open } for pid=61303 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.302:17306): avc: denied { read } for pid=69894 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823901.936:17275): avc: denied { read } for pid=886 comm="sssd" name="resolv.conf" dev="sda1" ino=394966 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.302:17308): avc: denied { map } for pid=69894 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823533.761:405): avc: denied { lock } for pid=7009 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.302:17307): avc: denied { lock } for pid=69894 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.848:15056): avc: denied { open } for pid=61475 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.119:15002): avc: denied { lock } for pid=61303 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823804.119:15003): avc: denied { map } for pid=61303 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.302:17306): avc: denied { open } for pid=69894 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139723] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139723/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4ad259ca06642bb8e630f0d3d0360a1 timeout expired in wait_until_healthy [7139724] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:57:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139724/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139724/ "2023-01-27T13:13:24.951341+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139725] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139725/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139725/ SELinux denials found on ubuntu@smithi006.front.sepia.ceph.com: ['type=AVC msg=audit(1674823629.943:419): avc: denied { read } for pid=6990 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823629.943:419): avc: denied { open } for pid=6990 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.432:15024): avc: denied { read } for pid=61333 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.432:15026): avc: denied { map } for pid=61333 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.077:15001): avc: denied { search } for pid=6900 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674823629.943:420): avc: denied { lock } for pid=6990 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.432:15024): avc: denied { open } for pid=61333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823902.432:15025): avc: denied { lock } for pid=61333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823629.943:421): avc: denied { map } for pid=6990 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139727] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139727/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139727/ "2023-01-27T13:02:23.230238+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139728] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 02:09:42 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139728/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139728/ SELinux denials found on ubuntu@smithi085.front.sepia.ceph.com: ['type=AVC msg=audit(1674826866.023:18132): avc: denied { read } for pid=292089 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823869.588:210): avc: denied { lock } for pid=2416 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823869.575:209): avc: denied { open } for pid=2416 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823869.599:211): avc: denied { map } for pid=2416 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826866.023:18134): avc: denied { map } for pid=292089 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823870.590:215): avc: denied { map } for pid=3703 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823869.575:209): avc: denied { read } for pid=2416 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826866.023:18132): avc: denied { open } for pid=292089 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823870.590:213): avc: denied { read } for pid=3703 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823870.590:213): avc: denied { open } for pid=3703 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674823870.590:214): avc: denied { lock } for pid=3703 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826862.657:18131): avc: denied { search } for pid=1542 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525248 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674826866.023:18133): avc: denied { lock } for pid=292089 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139731] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:30:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139731/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139731/ SELinux denials found on ubuntu@smithi114.front.sepia.ceph.com: ['type=AVC msg=audit(1674824298.937:15028): avc: denied { map } for pid=61356 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824298.937:15026): avc: denied { read } for pid=61356 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824945.293:19923): avc: denied { map } for pid=115545 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=393227 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824942.088:19912): avc: denied { search } for pid=6924 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674824298.588:15001): avc: denied { search } for pid=6924 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674824031.035:416): avc: denied { lock } for pid=6997 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824298.937:15027): avc: denied { lock } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824298.937:15026): avc: denied { open } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824031.035:417): avc: denied { map } for pid=6997 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824031.035:415): avc: denied { open } for pid=6997 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824031.035:415): avc: denied { read } for pid=6997 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824945.293:19922): avc: denied { lock } for pid=115545 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824945.293:19921): avc: denied { read } for pid=115545 comm="rpm" name="Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824945.293:19921): avc: denied { open } for pid=115545 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139732] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:27:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139732/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bddfb358dc734c91856ab7e5e2b60e99 Command failed (workunit test cephtool/test.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7139733] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139733/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139733/ SELinux denials found on ubuntu@smithi174.front.sepia.ceph.com: ['type=AVC msg=audit(1674824133.670:421): avc: denied { map } for pid=6988 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824400.923:15025): avc: denied { lock } for pid=61333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824133.670:419): avc: denied { open } for pid=6988 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824400.923:15024): avc: denied { read } for pid=61333 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824400.566:14997): avc: denied { search } for pid=6898 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674824400.923:15026): avc: denied { map } for pid=61333 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824133.670:420): avc: denied { lock } for pid=6988 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824133.670:419): avc: denied { read } for pid=6988 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824400.923:15024): avc: denied { open } for pid=61333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139734] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-bitmap} tasks/e2e} ----------------------------------------------------------------- time: 00:23:54 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139734/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139734/ SELinux denials found on ubuntu@smithi117.front.sepia.ceph.com: ['type=AVC msg=audit(1674824568.783:14075): avc: denied { map } for pid=77554 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824310.758:213): avc: denied { open } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824310.779:215): avc: denied { map } for pid=6344 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824567.950:14041): avc: denied { map } for pid=77401 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824567.950:14040): avc: denied { lock } for pid=77401 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824568.783:14074): avc: denied { lock } for pid=77554 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824568.783:14073): avc: denied { read } for pid=77554 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824310.769:214): avc: denied { lock } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824567.950:14039): avc: denied { open } for pid=77401 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824310.758:213): avc: denied { read } for pid=6344 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824567.950:14039): avc: denied { read } for pid=77401 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824568.783:14073): avc: denied { open } for pid=77554 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824566.794:14036): avc: denied { search } for pid=6228 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139735] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/master} ----------------------------------------------------------------- time: 00:05:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139735/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139735/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7c6e173bf2f4d1e8ffc3f550f2bc0f7 Command failed on smithi103 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7139736] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139736/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139736/ SELinux denials found on ubuntu@smithi072.front.sepia.ceph.com: ['type=AVC msg=audit(1674824615.455:14040): avc: denied { lock } for pid=77419 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824365.607:216): avc: denied { lock } for pid=6368 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824615.455:14039): avc: denied { open } for pid=77419 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824365.622:217): avc: denied { map } for pid=6368 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824615.455:14039): avc: denied { read } for pid=77419 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824615.455:14041): avc: denied { map } for pid=77419 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824365.597:215): avc: denied { read } for pid=6368 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824365.597:215): avc: denied { open } for pid=6368 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824615.100:14038): avc: denied { search } for pid=6255 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139737] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:22:22 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139737/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139737/ "2023-01-27T13:12:39.557669+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139738] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:27:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139738/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139738/ "2023-01-27T13:12:57.216528+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139740] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:18:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139740/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcdc396ba3fc48d0a3058f2b01200a5a timeout expired in wait_until_healthy [7139741] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/scrub_test} ----------------------------------------------------------------- time: 00:13:12 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139741/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139741/ "2023-01-27T13:17:00.845858+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139742] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:13:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139742/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139742/ SELinux denials found on ubuntu@smithi203.front.sepia.ceph.com: ['type=AVC msg=audit(1674824868.253:213): avc: denied { open } for pid=6362 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824868.265:214): avc: denied { lock } for pid=6362 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825113.374:14036): avc: denied { search } for pid=6249 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674824868.272:215): avc: denied { map } for pid=6362 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825237.384:16992): avc: denied { read } for pid=94016 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825113.725:14038): avc: denied { lock } for pid=77401 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825237.384:16992): avc: denied { open } for pid=94016 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825237.033:16961): avc: denied { read } for pid=724 comm="sssd" name="resolv.conf" dev="sda1" ino=525302 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825113.725:14037): avc: denied { open } for pid=77401 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825237.384:16994): avc: denied { map } for pid=94016 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825113.725:14039): avc: denied { map } for pid=77401 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825237.384:16993): avc: denied { lock } for pid=94016 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824868.253:213): avc: denied { read } for pid=6362 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825113.725:14037): avc: denied { read } for pid=77401 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139743] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:28:15 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139743/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139743/ "2023-01-27T13:16:15.145450+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139744] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:27:36 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139744/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139744/ "2023-01-27T13:22:29.946474+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139746] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:29 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139746/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139746/ SELinux denials found on ubuntu@smithi082.front.sepia.ceph.com: ['type=AVC msg=audit(1674825159.465:15024): avc: denied { read } for pid=61358 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825159.465:15024): avc: denied { open } for pid=61358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824893.675:419): avc: denied { read } for pid=7014 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825159.465:15025): avc: denied { lock } for pid=61358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824893.676:420): avc: denied { lock } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674824893.676:421): avc: denied { map } for pid=7014 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825159.119:15003): avc: denied { search } for pid=6924 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674824893.675:419): avc: denied { open } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674825159.466:15026): avc: denied { map } for pid=61358 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139748] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:18:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139748/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=005134961a994cfda16f104c1af353e3 timeout expired in wait_until_healthy [7139751] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139751/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139751/ "2023-01-27T13:22:48.659921+0000 mgr.y (mgr.4123) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139752] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:16:15 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139752/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139752/ "2023-01-27T13:24:59.425361+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139753] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139753/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3df2397bf03e4c18b9e79276fa83168f timeout expired in wait_until_healthy [7139754] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139754/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d84a2d4359f445f198fdc4bbc9f663ac Command failed (workunit test rados/test_librados_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_librados_build.sh' [7139755] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:29:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139755/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139755/ "2023-01-27T13:33:41.921618+0000 mon.e (mon.8) 4050 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139756] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:19:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139756/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139756/ "2023-01-27T13:27:31.429669+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139761] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:34:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139761/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139761/ SELinux denials found on ubuntu@smithi101.front.sepia.ceph.com: ['type=AVC msg=audit(1674826532.429:14037): avc: denied { open } for pid=77420 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826279.241:214): avc: denied { lock } for pid=6378 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826656.856:16990): avc: denied { read } for pid=94029 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826532.429:14037): avc: denied { read } for pid=77420 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826279.228:213): avc: denied { open } for pid=6378 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826279.228:213): avc: denied { read } for pid=6378 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826656.856:16991): avc: denied { lock } for pid=94029 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826656.492:16959): avc: denied { read } for pid=711 comm="sssd" name="resolv.conf" dev="sda1" ino=525302 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826532.429:14038): avc: denied { lock } for pid=77420 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826279.242:215): avc: denied { map } for pid=6378 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826531.210:14034): avc: denied { search } for pid=6265 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674826656.856:16992): avc: denied { map } for pid=94029 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826532.429:14039): avc: denied { map } for pid=77420 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826656.856:16990): avc: denied { open } for pid=94029 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139763] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:15:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139763/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139763/ SELinux denials found on ubuntu@smithi117.front.sepia.ceph.com: ['type=AVC msg=audit(1674826546.750:14036): avc: denied { lock } for pid=77317 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826295.488:214): avc: denied { lock } for pid=6341 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826295.498:215): avc: denied { map } for pid=6341 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826546.750:14037): avc: denied { map } for pid=77317 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826546.750:14035): avc: denied { read } for pid=77317 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826295.477:213): avc: denied { open } for pid=6341 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826295.477:213): avc: denied { read } for pid=6341 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826546.387:14034): avc: denied { search } for pid=6228 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674826546.750:14035): avc: denied { open } for pid=77317 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139765] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:18:08 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139765/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139765/ SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1674826457.408:15023): avc: denied { read } for pid=61360 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826181.032:418): avc: denied { open } for pid=7017 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826557.593:17276): avc: denied { read } for pid=886 comm="sssd" name="resolv.conf" dev="sda1" ino=394966 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826457.408:15024): avc: denied { lock } for pid=61360 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826557.957:17307): avc: denied { open } for pid=69895 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826557.957:17309): avc: denied { map } for pid=69895 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826557.957:17308): avc: denied { lock } for pid=69895 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826181.032:419): avc: denied { lock } for pid=7017 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826557.957:17307): avc: denied { read } for pid=69895 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826457.408:15023): avc: denied { open } for pid=61360 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826457.408:15025): avc: denied { map } for pid=61360 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826181.032:420): avc: denied { map } for pid=7017 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826457.050:15002): avc: denied { search } for pid=6927 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674826181.032:418): avc: denied { read } for pid=7017 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139766] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139766/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139766/ "2023-01-27T13:41:59.248855+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139768] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:19:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139768/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5640460bc9794bb58443508a6f8ab93a timeout expired in wait_until_healthy [7139769] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:25:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139769/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139769/ "2023-01-27T13:42:20.844125+0000 mgr.x (mgr.4120) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139770] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:27:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139770/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139770/ "2023-01-27T13:43:04.708331+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139772] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:47:14 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139772/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139772/ "2023-01-27T13:50:48.564291+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139773] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:29:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139773/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139773/ "2023-01-27T13:54:25.062138+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139774] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:21:55 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139774/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0f8e93c10564ff99f393a0952531960 timeout expired in wait_until_healthy [7139775] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:30:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139775/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139775/ "2023-01-27T13:48:14.299385+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139777] rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} ----------------------------------------------------------------- time: 02:28:24 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139777/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139777/ SELinux denials found on ubuntu@smithi137.front.sepia.ceph.com: ['type=AVC msg=audit(1674826917.107:15024): avc: denied { open } for pid=61337 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.107:18460): avc: denied { getattr } for pid=92087 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=394957 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826650.939:421): avc: denied { map } for pid=6994 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829805.623:18506): avc: denied { read } for pid=257686 comm="rpm" name="Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.106:18458): avc: denied { read write } for pid=92087 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=394956 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827757.779:18467): avc: denied { lock } for pid=94061 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827757.779:18466): avc: denied { open } for pid=94061 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826650.939:419): avc: denied { open } for pid=6994 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.499:18462): avc: denied { read } for pid=92196 comm="rpm" name="Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829805.623:18506): avc: denied { open } for pid=257686 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829802.264:18505): avc: denied { search } for pid=6904 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674827723.753:18465): avc: denied { read } for pid=92087 comm="rhsmcertd-worke" name="python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826650.939:420): avc: denied { lock } for pid=6994 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.107:18461): avc: denied { map } for pid=92087 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=394957 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.499:18463): avc: denied { lock } for pid=92196 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.499:18462): avc: denied { open } for pid=92196 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.499:18464): avc: denied { map } for pid=92196 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394938 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826650.939:419): avc: denied { read } for pid=6994 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.106:18458): avc: denied { open } for pid=92087 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=394956 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829805.623:18508): avc: denied { map } for pid=257686 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394938 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827757.779:18466): avc: denied { read } for pid=94061 comm="rpm" name="Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829805.623:18507): avc: denied { lock } for pid=257686 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394937 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.753:18465): avc: denied { open } for pid=92087 comm="rhsmcertd-worke" path="/etc/dnf/modules.d/python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827757.779:18468): avc: denied { map } for pid=94061 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394938 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826917.107:15026): avc: denied { map } for pid=61337 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827723.106:18459): avc: denied { lock } for pid=92087 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=394956 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826917.107:15025): avc: denied { lock } for pid=61337 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826917.107:15024): avc: denied { read } for pid=61337 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674826916.761:15003): avc: denied { search } for pid=6904 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139781] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:24:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139781/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139781/ "2023-01-27T14:00:08.664029+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139782] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:27:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139782/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139782/ SELinux denials found on ubuntu@smithi050.front.sepia.ceph.com: ['type=AVC msg=audit(1674827822.379:14035): avc: denied { search } for pid=6234 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674827567.202:214): avc: denied { open } for pid=6347 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827822.744:14036): avc: denied { read } for pid=77366 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827822.744:14036): avc: denied { open } for pid=77366 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827822.744:14038): avc: denied { map } for pid=77366 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827822.744:14037): avc: denied { lock } for pid=77366 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827567.219:216): avc: denied { map } for pid=6347 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827567.202:214): avc: denied { read } for pid=6347 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827567.219:215): avc: denied { lock } for pid=6347 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139783] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:26:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139783/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139783/ "2023-01-27T14:03:15.934604+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139785] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:44:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139785/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139785/ "2023-01-27T14:10:11.664698+0000 mon.a (mon.0) 851 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139786] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:28:40 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139786/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139786/ "2023-01-27T14:05:50.033152+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139787] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:30:55 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139787/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139787/ "2023-01-27T14:20:33.353771+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139789] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139789/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139789/ SELinux denials found on ubuntu@smithi073.front.sepia.ceph.com: ['type=AVC msg=audit(1674828195.183:14039): avc: denied { map } for pid=77403 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827950.295:213): avc: denied { open } for pid=6351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828195.183:14038): avc: denied { lock } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827950.327:215): avc: denied { map } for pid=6351 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828317.597:16992): avc: denied { read } for pid=94017 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828317.597:16992): avc: denied { open } for pid=94017 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828317.598:16994): avc: denied { map } for pid=94017 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828195.183:14037): avc: denied { read } for pid=77403 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828194.824:14036): avc: denied { search } for pid=6237 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674828195.183:14037): avc: denied { open } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828317.249:16961): avc: denied { read } for pid=695 comm="sssd" name="resolv.conf" dev="sda1" ino=525265 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828317.597:16993): avc: denied { lock } for pid=94017 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827950.312:214): avc: denied { lock } for pid=6351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674827950.295:213): avc: denied { read } for pid=6351 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139790] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:21:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139790/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a1abe644b6045098d9bf867cd248519 timeout expired in wait_until_healthy [7139791] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:59 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139791/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139791/ SELinux denials found on ubuntu@smithi202.front.sepia.ceph.com: ['type=AVC msg=audit(1674828084.139:15023): avc: denied { open } for pid=61373 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828179.671:17421): avc: denied { lock } for pid=70297 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828179.671:17420): avc: denied { open } for pid=70297 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828084.139:15025): avc: denied { map } for pid=61373 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828178.891:17337): avc: denied { open } for pid=70006 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828083.792:15002): avc: denied { search } for pid=6927 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674828178.891:17339): avc: denied { map } for pid=70006 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828178.891:17338): avc: denied { lock } for pid=70006 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828178.530:17306): avc: denied { read } for pid=890 comm="sssd" name="resolv.conf" dev="sda1" ino=394967 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828179.671:17422): avc: denied { map } for pid=70297 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828084.139:15023): avc: denied { read } for pid=61373 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828178.891:17337): avc: denied { read } for pid=70006 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828084.139:15024): avc: denied { lock } for pid=61373 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828179.671:17420): avc: denied { read } for pid=70297 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139792] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139792/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139792/ "2023-01-27T14:16:13.834386+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139795] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:13:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139795/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139795/ SELinux denials found on ubuntu@smithi012.front.sepia.ceph.com: ['type=AVC msg=audit(1674828733.113:17014): avc: denied { lock } for pid=94004 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525261 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828350.510:218): avc: denied { map } for pid=6325 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828733.113:17013): avc: denied { open } for pid=94004 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525261 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828351.226:220): avc: denied { open } for pid=6327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828733.113:17013): avc: denied { read } for pid=94004 comm="rpm" name="Packages" dev="sda1" ino=525261 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828604.602:14043): avc: denied { search } for pid=6125 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393279 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674828350.493:216): avc: denied { open } for pid=6325 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828605.891:14048): avc: denied { map } for pid=77373 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828350.493:216): avc: denied { read } for pid=6325 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828605.891:14046): avc: denied { read } for pid=77373 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828351.226:222): avc: denied { map } for pid=6327 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828350.509:217): avc: denied { lock } for pid=6325 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828351.226:221): avc: denied { lock } for pid=6327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828605.891:14046): avc: denied { open } for pid=77373 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828605.891:14047): avc: denied { lock } for pid=77373 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828351.226:220): avc: denied { read } for pid=6327 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828732.747:16982): avc: denied { read } for pid=703 comm="sssd" name="resolv.conf" dev="sda1" ino=525293 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828733.113:17015): avc: denied { map } for pid=94004 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525262 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139796] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:37:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139796/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c09f6cf537944df6a7c021017c30883e timeout expired in wait_until_healthy [7139799] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:14:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139799/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139799/ "2023-01-27T14:25:09.604259+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139801] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:26:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139801/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e291421a694e4228a1758a881058bac7 timeout expired in wait_until_healthy [7139802] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:22:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139802/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139802/ SELinux denials found on ubuntu@smithi101.front.sepia.ceph.com: ['type=AVC msg=audit(1674829245.165:14040): avc: denied { lock } for pid=77388 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829245.165:14039): avc: denied { open } for pid=77388 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829367.152:16993): avc: denied { lock } for pid=93999 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829244.172:14036): avc: denied { search } for pid=6231 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674829245.165:14039): avc: denied { read } for pid=77388 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829367.152:16992): avc: denied { read } for pid=93999 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829245.165:14041): avc: denied { map } for pid=77388 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829366.801:16961): avc: denied { read } for pid=689 comm="sssd" name="resolv.conf" dev="sda1" ino=525265 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829367.152:16994): avc: denied { map } for pid=93999 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828995.698:213): avc: denied { read } for pid=6344 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828995.698:213): avc: denied { open } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828995.715:215): avc: denied { map } for pid=6344 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828995.711:214): avc: denied { lock } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829367.152:16992): avc: denied { open } for pid=93999 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139803] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:17:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139803/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139803/ SELinux denials found on ubuntu@smithi175.front.sepia.ceph.com: ['type=AVC msg=audit(1674828821.236:419): avc: denied { open } for pid=6993 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15024): avc: denied { map } for pid=61334 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15023): avc: denied { lock } for pid=61334 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828821.236:420): avc: denied { lock } for pid=6993 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15022): avc: denied { open } for pid=61334 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.124:15001): avc: denied { search } for pid=6903 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674828821.236:421): avc: denied { map } for pid=6993 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674829085.469:15022): avc: denied { read } for pid=61334 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674828821.236:419): avc: denied { read } for pid=6993 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139804] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:26:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139804/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139804/ "2023-01-27T14:25:10.391125+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139806] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:27:51 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139806/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2812afdafa2c417794c709149792f643 Command failed (workunit test cephtool/test.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [7139807] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139807/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139807/ "2023-01-27T14:36:34.392811+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139808] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:18:51 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139808/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3171d2bce7774c769b44368f8c062f1f timeout expired in wait_until_healthy [7139810] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:25:27 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139810/ "2023-01-27T14:58:32.678533+0000 mon.r (mon.17) 44 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139812] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} ----------------------------------------------------------------- time: 00:20:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139812/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=407189f60e344ee98701d3f6d0bd3b4a Test failure: test_token_ttl (tasks.mgr.dashboard.test_auth.AuthTest) [7139813] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/crash} ----------------------------------------------------------------- time: 00:15:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139813/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139813/ "2023-01-27T14:52:24.403093+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139814] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139814/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139814/ SELinux denials found on ubuntu@smithi008.front.sepia.ceph.com: ['type=AVC msg=audit(1674831102.548:17058): avc: denied { lock } for pid=94277 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830721.529:210): avc: denied { open } for pid=2228 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831102.548:17059): avc: denied { map } for pid=94277 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830722.343:215): avc: denied { lock } for pid=3438 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830721.537:211): avc: denied { lock } for pid=2228 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831101.739:16990): avc: denied { lock } for pid=93970 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830722.343:216): avc: denied { map } for pid=3438 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831101.739:16989): avc: denied { open } for pid=93970 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831101.370:16958): avc: denied { read } for pid=694 comm="sssd" name="resolv.conf" dev="sda1" ino=655363 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830721.529:210): avc: denied { read } for pid=2228 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830722.343:214): avc: denied { open } for pid=3438 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831102.548:17057): avc: denied { read } for pid=94277 comm="rpm" name="Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831101.739:16989): avc: denied { read } for pid=93970 comm="rpm" name="Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830722.343:214): avc: denied { read } for pid=3438 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831101.739:16991): avc: denied { map } for pid=93970 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831102.548:17057): avc: denied { open } for pid=94277 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830721.541:212): avc: denied { map } for pid=2228 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139815] rados/rest/{mgr-restful supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:16:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139815/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139815/ SELinux denials found on ubuntu@smithi007.front.sepia.ceph.com: ['type=AVC msg=audit(1674830550.983:421): avc: denied { map } for pid=7011 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830924.790:17310): avc: denied { read } for pid=69877 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830924.790:17310): avc: denied { open } for pid=69877 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830550.983:420): avc: denied { lock } for pid=7011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830824.787:15004): avc: denied { open } for pid=61294 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830824.434:14981): avc: denied { search } for pid=6921 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674830824.787:15004): avc: denied { read } for pid=61294 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830824.787:15005): avc: denied { lock } for pid=61294 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830924.790:17312): avc: denied { map } for pid=69877 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830550.983:419): avc: denied { read } for pid=7011 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830824.787:15006): avc: denied { map } for pid=61294 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830924.422:17275): avc: denied { read } for pid=880 comm="sssd" name="resolv.conf" dev="sda1" ino=394966 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830924.790:17311): avc: denied { lock } for pid=69877 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830550.983:419): avc: denied { open } for pid=7011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139816] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:05:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139816/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5b5fc482f4c4478afebb5df2afa4013 Command failed on smithi026 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7139818] rados/standalone/{supported-random-distro$/{centos_8} workloads/c2c} ----------------------------------------------------------------- time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139818/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139818/ SELinux denials found on ubuntu@smithi203.front.sepia.ceph.com: ['type=AVC msg=audit(1674830986.061:213): avc: denied { read } for pid=6358 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831231.850:14039): avc: denied { read } for pid=77403 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830986.061:213): avc: denied { open } for pid=6358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830986.073:214): avc: denied { lock } for pid=6358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831230.459:14036): avc: denied { search } for pid=6245 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674831231.850:14040): avc: denied { lock } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831231.850:14039): avc: denied { open } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831231.850:14041): avc: denied { map } for pid=77403 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674830986.079:215): avc: denied { map } for pid=6358 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139819] rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:37:51 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e100578167fb4134821088d4e38d5bc5 Command failed (workunit test cls/test_cls_rbd.sh) on smithi101 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7139820] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} ----------------------------------------------------------------- time: 00:25:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139820/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139820/ "2023-01-27T15:12:34.106870+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139821] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:21:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139821/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139821/ "2023-01-27T15:01:57.220534+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139822] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139822/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bcce339af2e42b29a24c8c69c3d5736 Command failed (workunit test cephadm/test_cephadm.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7139824] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:26:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139824/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139824/ "2023-01-27T15:06:22.890213+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139825] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139825/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139825/ "2023-01-27T15:07:28.713020+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139827] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:13:15 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139827/ SELinux denials found on ubuntu@smithi050.front.sepia.ceph.com: ['type=AVC msg=audit(1674831299.518:214): avc: denied { map } for pid=3592 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831299.518:212): avc: denied { open } for pid=3592 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831299.518:213): avc: denied { lock } for pid=3592 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831298.590:209): avc: denied { lock } for pid=2450 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831298.565:208): avc: denied { read } for pid=2450 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831299.518:212): avc: denied { read } for pid=3592 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831298.565:208): avc: denied { open } for pid=2450 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831298.590:210): avc: denied { map } for pid=2450 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139829] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:22:12 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139829/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9c4fae48734499f967c7b53461dc760 timeout expired in wait_until_healthy [7139831] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:18:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139831/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139831/ "2023-01-27T15:11:28.063312+0000 mon.a (mon.0) 961 : cluster [WRN] Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139834] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=caa81637ab4f4c9abf7c6d37d1a44f58 timeout expired in wait_until_healthy [7139835] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:32:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139835/ "2023-01-27T15:07:51.451432+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139836] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:16:17 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139836/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139836/ SELinux denials found on ubuntu@smithi047.front.sepia.ceph.com: ['type=AVC msg=audit(1674831778.132:14039): avc: denied { map } for pid=77380 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831908.730:16992): avc: denied { read } for pid=93998 comm="rpm" name="Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831908.364:16961): avc: denied { read } for pid=693 comm="sssd" name="resolv.conf" dev="sda1" ino=525300 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831515.521:213): avc: denied { open } for pid=6321 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831908.731:16994): avc: denied { map } for pid=93998 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525268 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831778.132:14037): avc: denied { read } for pid=77380 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831908.730:16993): avc: denied { lock } for pid=93998 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831515.538:215): avc: denied { map } for pid=6321 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831778.132:14038): avc: denied { lock } for pid=77380 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831515.521:213): avc: denied { read } for pid=6321 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831778.132:14037): avc: denied { open } for pid=77380 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831515.538:214): avc: denied { lock } for pid=6321 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831908.730:16992): avc: denied { open } for pid=93998 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831777.749:14036): avc: denied { search } for pid=6208 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139837] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:22:56 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139837/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139837/ "2023-01-27T15:11:00.130625+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139839] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:26:30 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139839/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139839/ "2023-01-27T15:11:56.792939+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139840] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139840/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139840/ "2023-01-27T15:14:37.688101+0000 mgr.x (mgr.10017) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139841] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=742a6de69fdd4c578e8ad7f2b29c19f5 timeout expired in wait_until_healthy [7139843] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:23:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139843/ "2023-01-27T15:16:26.870629+0000 mon.a (mon.0) 1239 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139846] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:23:47 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139846/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139846/ "2023-01-27T15:14:01.212588+0000 mon.a (mon.0) 321 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139847] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:50 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139847/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ffd37e82af64e598c8c6f38e03b9174 Command failed (workunit test cephadm/test_repos.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7139849] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_8} tasks/failover} ----------------------------------------------------------------- time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139849/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139849/ "2023-01-27T15:12:07.041929+0000 mgr.z (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139850] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 00:32:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139850/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139850/ "2023-01-27T15:20:13.737589+0000 mon.a (mon.0) 1185 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139853] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:24:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139853/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139853/ "2023-01-27T15:11:14.254606+0000 mgr.x (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139854] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139854/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=685ce59917a045608a4e747aa9c59bbe Command failed on smithi088 with status 1: 'sudo yum install -y kernel' [7139855] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:05:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139855/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139855/ "2023-01-27T15:22:50.296048+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139856] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139856/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15eb4579fba64e57b4cac5da602a13c1 Command failed on smithi097 with status 1: 'sudo yum install -y kernel' [7139857] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:22:14 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139857/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=899a25f2638c4933bd20be40db8e76a1 timeout expired in wait_until_healthy [7139861] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139861/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139861/ SELinux denials found on ubuntu@smithi110.front.sepia.ceph.com: ['type=AVC msg=audit(1674831787.948:14038): avc: denied { read } for pid=77307 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831787.948:14038): avc: denied { open } for pid=77307 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831531.330:216): avc: denied { map } for pid=6346 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831531.330:215): avc: denied { lock } for pid=6346 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831787.948:14039): avc: denied { lock } for pid=77307 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831531.312:214): avc: denied { read } for pid=6346 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831531.312:214): avc: denied { open } for pid=6346 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831787.527:14035): avc: denied { search } for pid=6233 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674831787.948:14040): avc: denied { map } for pid=77307 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139863] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139863/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b648467a03bf4310a9ef465b3fc55cf6 timeout expired in wait_until_healthy [7139864] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:21:01 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139864/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4261d1fb9d204f01a41eff1b77494b26 Command failed (workunit test rados/test_crash.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh' [7139865] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:28:17 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139865/ "2023-01-27T15:15:35.322877+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139867] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139867/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139867/ SELinux denials found on ubuntu@smithi098.front.sepia.ceph.com: ['type=AVC msg=audit(1674832322.931:16987): avc: denied { lock } for pid=93988 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832323.732:17052): avc: denied { map } for pid=94295 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832323.732:17051): avc: denied { lock } for pid=94295 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.870:211): avc: denied { read } for pid=2851 comm="rpm" name="Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.870:211): avc: denied { open } for pid=2851 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832322.931:16988): avc: denied { map } for pid=93988 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832323.732:17050): avc: denied { read } for pid=94295 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.875:212): avc: denied { map } for pid=2851 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832322.931:16986): avc: denied { open } for pid=93988 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.870:210): avc: denied { lock } for pid=2851 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832322.931:16986): avc: denied { read } for pid=93988 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.858:209): avc: denied { open } for pid=2851 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832323.732:17050): avc: denied { open } for pid=94295 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831936.858:209): avc: denied { read } for pid=2851 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832322.567:16955): avc: denied { read } for pid=695 comm="sssd" name="resolv.conf" dev="sda1" ino=525297 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1'] [7139868] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:30:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91ec0aa05cbe463e8da05d3416603588 Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS) [7139871] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139871/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139871/ SELinux denials found on ubuntu@smithi016.front.sepia.ceph.com: ['type=AVC msg=audit(1674832059.486:15024): avc: denied { map } for pid=61354 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:419): avc: denied { open } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:420): avc: denied { lock } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.309:17275): avc: denied { read } for pid=899 comm="sssd" name="resolv.conf" dev="sda1" ino=393262 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15022): avc: denied { read } for pid=61354 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:419): avc: denied { read } for pid=7013 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15022): avc: denied { open } for pid=61354 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17379): avc: denied { map } for pid=70126 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=393227 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17377): avc: denied { read } for pid=70126 comm="rpm" name="Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17308): avc: denied { map } for pid=69887 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=393227 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831769.066:421): avc: denied { map } for pid=7013 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17377): avc: denied { open } for pid=70126 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17306): avc: denied { open } for pid=69887 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17307): avc: denied { lock } for pid=69887 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.486:15023): avc: denied { lock } for pid=61354 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832164.677:17306): avc: denied { read } for pid=69887 comm="rpm" name="Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832165.439:17378): avc: denied { lock } for pid=70126 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=393226 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832059.133:15001): avc: denied { search } for pid=6923 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139872] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139872/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139872/ SELinux denials found on ubuntu@smithi086.front.sepia.ceph.com: ['type=AVC msg=audit(1674832224.494:14040): avc: denied { lock } for pid=77408 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832347.952:16995): avc: denied { read } for pid=94019 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831963.345:213): avc: denied { lock } for pid=6361 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832223.044:14036): avc: denied { search } for pid=6248 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674832347.581:16964): avc: denied { read } for pid=714 comm="sssd" name="resolv.conf" dev="sda1" ino=525297 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832225.262:14059): avc: denied { read } for pid=77483 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832225.262:14060): avc: denied { lock } for pid=77483 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832347.952:16997): avc: denied { map } for pid=94019 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831963.328:212): avc: denied { read } for pid=6361 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831963.350:214): avc: denied { map } for pid=6361 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674831963.328:212): avc: denied { open } for pid=6361 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832347.952:16995): avc: denied { open } for pid=94019 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832225.262:14061): avc: denied { map } for pid=77483 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832224.494:14039): avc: denied { read } for pid=77408 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832224.494:14041): avc: denied { map } for pid=77408 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832224.494:14039): avc: denied { open } for pid=77408 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832347.952:16996): avc: denied { lock } for pid=94019 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832225.262:14059): avc: denied { open } for pid=77483 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139873] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:29:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139873/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139873/ "2023-01-27T15:16:18.502018+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139874] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139874/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139874/ "2023-01-27T15:18:36.310066+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139875] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:22:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fee8e57c4bad43919684a87da6894af5 timeout expired in wait_until_healthy [7139876] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139876/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139876/ SELinux denials found on ubuntu@smithi039.front.sepia.ceph.com: ['type=AVC msg=audit(1674832701.770:18361): avc: denied { lock } for pid=75770 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.369:18358): avc: denied { getattr } for pid=75695 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=394950 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832715.355:18754): avc: denied { lock } for pid=76681 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832050.398:420): avc: denied { open } for pid=6976 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832715.355:18753): avc: denied { open } for pid=76681 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832702.016:18363): avc: denied { open } for pid=75695 comm="rhsmcertd-worke" path="/etc/dnf/modules.d/python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832335.286:15000): avc: denied { search } for pid=6886 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674832715.355:18753): avc: denied { read } for pid=76681 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832050.398:421): avc: denied { lock } for pid=6976 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832715.355:18755): avc: denied { map } for pid=76681 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.369:18356): avc: denied { read write } for pid=75695 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=394949 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832050.398:420): avc: denied { read } for pid=6976 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832335.648:15027): avc: denied { map } for pid=61319 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.769:18360): avc: denied { read } for pid=75770 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832702.016:18363): avc: denied { read } for pid=75695 comm="rhsmcertd-worke" name="python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832335.648:15026): avc: denied { lock } for pid=61319 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.369:18356): avc: denied { open } for pid=75695 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=394949 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.770:18362): avc: denied { map } for pid=75770 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.369:18359): avc: denied { map } for pid=75695 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=394950 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832335.648:15025): avc: denied { open } for pid=61319 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832335.648:15025): avc: denied { read } for pid=61319 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.369:18357): avc: denied { lock } for pid=75695 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=394949 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832701.769:18360): avc: denied { open } for pid=75770 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832050.398:422): avc: denied { map } for pid=6976 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139877] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:27:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139877/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139877/ "2023-01-27T15:19:07.006671+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139878] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139878/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39c6adf1ba2648339544ac04acf9da8b Cannot connect to remote host smithi097 [7139880] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139880/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01016bf1fdc34dafab1396aa8388eacc [Errno None] Unable to connect to port 22 on 172.21.15.88 [7139881] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:19:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139881/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139881/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ad47019c9f14ba4952e37309707548c timeout expired in wait_until_healthy [7139882] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:16:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139882/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139882/ "2023-01-27T15:26:02.006896+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139883] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{filestore-xfs} supported-random-distro$/{centos_8} tasks/insights} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139883/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139883/ "2023-01-27T15:26:05.163750+0000 mgr.z (mgr.4123) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139885] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_striper} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139885/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139885/ "2023-01-27T15:25:13.009858+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139887] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139887/ SELinux denials found on ubuntu@smithi129.front.sepia.ceph.com: ['type=AVC msg=audit(1674832842.599:15025): avc: denied { lock } for pid=61369 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832842.246:14999): avc: denied { search } for pid=6935 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674832842.599:15024): avc: denied { open } for pid=61369 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832842.599:15026): avc: denied { map } for pid=61369 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832842.599:15024): avc: denied { read } for pid=61369 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139888] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} ----------------------------------------------------------------- time: 00:24:52 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139888/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d303db7317c5403c9e754eecf1a891a2 Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test- erasure-code.sh' [7139889] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:09:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7079f9adcf8c4a6c9c410239bde0bd78 {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi088.front.sepia.ceph.com,172.21.15.88' (ECDSA) to the list of known hosts.\r\nubuntu@smithi088.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive)."}} [7139891] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:16:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139891/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139891/ SELinux denials found on ubuntu@smithi121.front.sepia.ceph.com: ['type=AVC msg=audit(1674833189.802:16993): avc: denied { open } for pid=94021 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832815.392:215): avc: denied { open } for pid=6365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833189.802:16994): avc: denied { lock } for pid=94021 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833189.802:16995): avc: denied { map } for pid=94021 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525268 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832815.404:216): avc: denied { lock } for pid=6365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833066.460:14042): avc: denied { map } for pid=77411 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833189.431:16962): avc: denied { read } for pid=691 comm="sssd" name="resolv.conf" dev="sda1" ino=525299 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832815.408:217): avc: denied { map } for pid=6365 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833066.460:14041): avc: denied { lock } for pid=77411 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833066.460:14040): avc: denied { open } for pid=77411 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832815.392:215): avc: denied { read } for pid=6365 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833189.802:16993): avc: denied { read } for pid=94021 comm="rpm" name="Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833066.460:14040): avc: denied { read } for pid=77411 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833065.313:14037): avc: denied { search } for pid=6252 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139892] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:48:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139892/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139892/ "2023-01-27T15:31:05.958728+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139894] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:24:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139894/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139894/ SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1674833130.822:14037): avc: denied { read } for pid=77365 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833130.822:14037): avc: denied { open } for pid=77365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.773:213): avc: denied { map } for pid=5861 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.746:211): avc: denied { read } for pid=5861 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.770:212): avc: denied { lock } for pid=5861 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833130.823:14039): avc: denied { map } for pid=77365 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833129.565:14034): avc: denied { search } for pid=4750 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393279 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674833130.823:14038): avc: denied { lock } for pid=77365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674832870.746:211): avc: denied { open } for pid=5861 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139895] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:05:45 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139895/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c980568b718747c38fc4984756682582 Command failed on smithi042 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7139896] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:45 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139896/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139896/ SELinux denials found on ubuntu@smithi053.front.sepia.ceph.com: ['type=AVC msg=audit(1674833013.112:15003): avc: denied { search } for pid=6903 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262150 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674833013.469:15024): avc: denied { read } for pid=61327 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15025): avc: denied { lock } for pid=61327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15026): avc: denied { map } for pid=61327 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833013.469:15024): avc: denied { open } for pid=61327 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139897] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 01:02:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139897/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139897/ "2023-01-27T15:32:34.996856+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139899] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:28:40 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139899/ SELinux denials found on ubuntu@smithi047.front.sepia.ceph.com: ['type=AVC msg=audit(1674836768.155:54561): avc: denied { open } for pid=507077 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836768.181:54563): avc: denied { map } for pid=507077 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836762.854:54536): avc: denied { search } for pid=6232 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674836768.155:54561): avc: denied { read } for pid=507077 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836769.550:54566): avc: denied { lock } for pid=507358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836769.550:54565): avc: denied { read } for pid=507358 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836768.180:54562): avc: denied { lock } for pid=507077 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836769.550:54567): avc: denied { map } for pid=507358 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836769.550:54565): avc: denied { open } for pid=507358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139900] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:22:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c94bf5d853ee4f0ea89916a1981e99e8 timeout expired in wait_until_healthy [7139901] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:26:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139901/ "2023-01-27T15:38:52.656719+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139902] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:37:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139902/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139902/ "2023-01-27T15:44:21.327439+0000 mon.a (mon.0) 1382 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139903] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 02:30:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139903/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139903/ "2023-01-27T15:41:24.657419+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139904] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:23:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139904/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139904/ "2023-01-27T15:42:11.811751+0000 mon.a (mon.0) 847 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139906] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/rgw 3-final} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139906/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139906/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5466592466d4e7c994bbbdc7611fa94 timeout expired in wait_until_healthy [7139908] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139908/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139908/ "2023-01-27T15:41:39.786893+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139909] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} ----------------------------------------------------------------- time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139909/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139909/ "2023-01-27T15:48:39.517991+0000 mon.a (mon.0) 154 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139910] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:24:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139910/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139910/ "2023-01-27T15:44:54.515668+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139911] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:26:50 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139911/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139911/ SELinux denials found on ubuntu@smithi073.front.sepia.ceph.com: ['type=AVC msg=audit(1674833763.647:14037): avc: denied { lock } for pid=77410 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833501.056:215): avc: denied { map } for pid=6370 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833763.647:14036): avc: denied { read } for pid=77410 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833501.049:214): avc: denied { lock } for pid=6370 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833501.033:213): avc: denied { read } for pid=6370 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833501.033:213): avc: denied { open } for pid=6370 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833763.647:14036): avc: denied { open } for pid=77410 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833763.647:14038): avc: denied { map } for pid=77410 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833763.291:14035): avc: denied { search } for pid=6257 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139912] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139912/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25679d3909604c8fac67141bfc800ace Command failed (workunit test rados/test_librados_build.sh) on smithi111 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_librados_build.sh' [7139913] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:30:59 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139913/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139913/ "2023-01-27T15:53:59.429266+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139917] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:26:32 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139917/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139917/ "2023-01-27T15:44:17.055795+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139919] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139919/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139919/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e9cdca0fbf1437b82eff8c1b001b145 timeout expired in wait_until_healthy [7139921] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:37:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139921/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139921/ "2023-01-27T15:51:28.747125+0000 mon.a (mon.0) 1245 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139922] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139922/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139922/ SELinux denials found on ubuntu@smithi101.front.sepia.ceph.com: ['type=AVC msg=audit(1674833800.219:15022): avc: denied { open } for pid=61351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833899.534:17308): avc: denied { map } for pid=69878 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833528.727:420): avc: denied { lock } for pid=7011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833528.727:421): avc: denied { map } for pid=7011 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833899.534:17307): avc: denied { lock } for pid=69878 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833800.219:15023): avc: denied { lock } for pid=61351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833800.219:15022): avc: denied { read } for pid=61351 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833800.219:15024): avc: denied { map } for pid=61351 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833899.188:17275): avc: denied { read } for pid=901 comm="sssd" name="resolv.conf" dev="sda1" ino=394966 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833528.726:419): avc: denied { open } for pid=7011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833899.534:17306): avc: denied { open } for pid=69878 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833528.726:419): avc: denied { read } for pid=7011 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833899.534:17306): avc: denied { read } for pid=69878 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833799.870:14997): avc: denied { search } for pid=6921 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139923] rados/standalone/{supported-random-distro$/{centos_8} workloads/mgr} ----------------------------------------------------------------- time: 00:17:52 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139923/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139923/ SELinux denials found on ubuntu@smithi100.front.sepia.ceph.com: ['type=AVC msg=audit(1674834110.597:17060): avc: denied { map } for pid=94316 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833727.944:215): avc: denied { read } for pid=6355 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833983.510:14037): avc: denied { read } for pid=77392 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833983.150:14036): avc: denied { search } for pid=6242 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674833727.956:216): avc: denied { lock } for pid=6355 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833983.510:14037): avc: denied { open } for pid=77392 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833727.961:217): avc: denied { map } for pid=6355 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833983.510:14039): avc: denied { map } for pid=77392 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833727.944:215): avc: denied { open } for pid=6355 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834110.597:17059): avc: denied { lock } for pid=94316 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834109.802:16996): avc: denied { map } for pid=94010 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834110.597:17058): avc: denied { open } for pid=94316 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834109.802:16994): avc: denied { read } for pid=94010 comm="rpm" name="Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834109.802:16995): avc: denied { lock } for pid=94010 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834109.455:16963): avc: denied { read } for pid=698 comm="sssd" name="resolv.conf" dev="sda1" ino=395818 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834110.597:17058): avc: denied { read } for pid=94316 comm="rpm" name="Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833983.510:14038): avc: denied { lock } for pid=77392 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834109.802:16994): avc: denied { open } for pid=94010 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139925] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:36:36 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139925/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139925/ "2023-01-27T15:46:11.838801+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139926] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:25:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139926/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139926/ "2023-01-27T15:46:35.040161+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139927] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:27:23 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139927/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db0eef34b88340d4b635377dc53b1c09 timeout expired in wait_until_healthy [7139928] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:23:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139928/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139928/ "2023-01-27T15:48:55.715106+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139930] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:27:54 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139930/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139930/ "2023-01-27T15:52:45.444057+0000 mon.a (mon.0) 270 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139931] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 02:15:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139931/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139931/ "2023-01-27T15:52:34.539718+0000 mon.a (mon.0) 91 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139933] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:07:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139933/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139933/ sentry: https://sentry.ceph.com/organizations/ceph/?query=008836295c36412794aafc5bbb363c7d {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 12, 'changed': True, 'cmd': ['subscription-manager', 'register', '-- activationkey=testnode', '--org=Ceph', '--name=smithi088', '--force'], 'delta': '0:00:01.535748', 'end': '2023-01-27 15:41:48.803096', 'failed_when_result': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager register --activationkey=testnode --org=Ceph --name=smithi088 --force', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 70, 'start': '2023-01-27 15:41:47.267348', 'stderr': 'HTTP error (404 - Not Found): Consumer with id 280720d2-f2e3-46aa-bee7-a0de62800d04 could not be found.', 'stderr_lines': ['HTTP error (404 - Not Found): Consumer with id 280720d2-f2e3-46aa- bee7-a0de62800d04 could not be found.'], 'stdout': '', 'stdout_lines': []}} [7139934] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} ----------------------------------------------------------------- time: 00:21:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139934/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139934/ "2023-01-27T15:50:06.055776+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139935] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139935/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139935/ SELinux denials found on ubuntu@smithi154.front.sepia.ceph.com: ['type=AVC msg=audit(1674834141.449:17276): avc: denied { read } for pid=888 comm="sssd" name="resolv.conf" dev="sda1" ino=394966 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833765.835:418): avc: denied { open } for pid=7054 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833765.835:419): avc: denied { lock } for pid=7054 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834041.982:14996): avc: denied { search } for pid=6964 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834042.332:15024): avc: denied { lock } for pid=61396 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834042.332:15025): avc: denied { map } for pid=61396 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834141.806:17308): avc: denied { lock } for pid=69930 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833765.835:418): avc: denied { read } for pid=7054 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834141.806:17307): avc: denied { read } for pid=69930 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834141.806:17309): avc: denied { map } for pid=69930 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834141.806:17307): avc: denied { open } for pid=69930 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833765.835:420): avc: denied { map } for pid=7054 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834042.332:15023): avc: denied { open } for pid=61396 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834042.332:15023): avc: denied { read } for pid=61396 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139937] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:39:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139937/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139937/ "2023-01-27T15:56:12.503228+0000 mon.a (mon.0) 278 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139941] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:20:29 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139941/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1e9014f467f4fefa62778eb132fe9fc timeout expired in wait_until_healthy [7139942] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139942/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139942/ SELinux denials found on ubuntu@smithi114.front.sepia.ceph.com: ['type=AVC msg=audit(1674834508.461:16990): avc: denied { read } for pid=94002 comm="rpm" name="Packages" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834125.579:214): avc: denied { map } for pid=6346 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834383.265:14035): avc: denied { read } for pid=77386 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834383.265:14036): avc: denied { lock } for pid=77386 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834125.560:213): avc: denied { lock } for pid=6346 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834383.265:14035): avc: denied { open } for pid=77386 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834508.101:16959): avc: denied { read } for pid=700 comm="sssd" name="resolv.conf" dev="sda1" ino=395818 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834508.461:16990): avc: denied { open } for pid=94002 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834508.461:16991): avc: denied { lock } for pid=94002 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834382.919:14034): avc: denied { search } for pid=6233 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834125.551:212): avc: denied { open } for pid=6346 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834508.461:16992): avc: denied { map } for pid=94002 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834125.551:212): avc: denied { read } for pid=6346 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834383.265:14037): avc: denied { map } for pid=77386 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139943] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139943/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139943/ SELinux denials found on ubuntu@smithi110.front.sepia.ceph.com: ['type=AVC msg=audit(1674834333.229:17307): avc: denied { read } for pid=889 comm="sssd" name="resolv.conf" dev="sda1" ino=394967 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834333.579:17339): avc: denied { lock } for pid=69994 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834235.593:15026): avc: denied { map } for pid=61351 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833962.272:419): avc: denied { read } for pid=7007 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833962.272:420): avc: denied { lock } for pid=7007 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834235.593:15024): avc: denied { open } for pid=61351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834235.593:15025): avc: denied { lock } for pid=61351 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834235.247:15003): avc: denied { search } for pid=6916 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834333.578:17338): avc: denied { read } for pid=69994 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834333.578:17338): avc: denied { open } for pid=69994 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833962.272:419): avc: denied { open } for pid=7007 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674833962.272:421): avc: denied { map } for pid=7007 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834333.579:17340): avc: denied { map } for pid=69994 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834235.593:15024): avc: denied { read } for pid=61351 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139944] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:27:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139944/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139944/ "2023-01-27T15:52:34.789267+0000 mgr.x (mgr.4123) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139945] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139945/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139945/ "2023-01-27T15:54:58.296779+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139946] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} ----------------------------------------------------------------- time: 00:14:38 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139946/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139946/ SELinux denials found on ubuntu@smithi117.front.sepia.ceph.com: ['type=AVC msg=audit(1674834314.800:15007): avc: denied { search } for pid=6897 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834031.239:427): avc: denied { map } for pid=6987 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834031.239:425): avc: denied { read } for pid=6987 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834315.146:15028): avc: denied { open } for pid=61328 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834031.239:425): avc: denied { open } for pid=6987 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834031.239:426): avc: denied { lock } for pid=6987 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834315.146:15028): avc: denied { read } for pid=61328 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834315.146:15029): avc: denied { lock } for pid=61328 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834315.146:15030): avc: denied { map } for pid=61328 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139947] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139947/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139947/ SELinux denials found on ubuntu@smithi119.front.sepia.ceph.com: ['type=AVC msg=audit(1674834558.632:17008): avc: denied { lock } for pid=94022 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834434.004:14039): avc: denied { search } for pid=6230 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834183.448:219): avc: denied { map } for pid=6343 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834434.357:14041): avc: denied { lock } for pid=77383 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834183.442:218): avc: denied { lock } for pid=6343 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834434.357:14040): avc: denied { read } for pid=77383 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834434.357:14042): avc: denied { map } for pid=77383 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834183.429:217): avc: denied { open } for pid=6343 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834558.632:17007): avc: denied { open } for pid=94022 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834558.275:16976): avc: denied { read } for pid=686 comm="sssd" name="resolv.conf" dev="sda1" ino=525298 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834434.357:14040): avc: denied { open } for pid=77383 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834183.429:217): avc: denied { read } for pid=6343 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834558.632:17007): avc: denied { read } for pid=94022 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834558.632:17009): avc: denied { map } for pid=94022 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139948] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:22:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139948/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77de6be34d384764ac3288d43433046e timeout expired in wait_until_healthy [7139949] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:18:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139949/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139949/ "2023-01-27T15:52:40.381044+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139950] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:25:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139950/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139950/ "2023-01-27T15:55:46.403770+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139953] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:12:30 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139953/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=230572d068204a59930340c219cd0590 Command failed (workunit test cephadm/test_cephadm.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0c8245b45314ecfe0b7cf39532aecb29f18f697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7139954] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:38:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139954/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d90ba19d9a3459d99e0167a1af052bf timeout expired in wait_until_healthy [7139955] rados/standalone/{supported-random-distro$/{rhel_8} workloads/misc} ----------------------------------------------------------------- time: 01:08:59 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139955/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139955/ SELinux denials found on ubuntu@smithi064.front.sepia.ceph.com: ['type=AVC msg=audit(1674835022.393:17349): avc: denied { open } for pid=70007 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835986.078:18491): avc: denied { lock } for pid=105146 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835982.696:18489): avc: denied { search } for pid=6910 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834922.197:14998): avc: denied { search } for pid=6910 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834644.311:420): avc: denied { open } for pid=7000 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835022.393:17350): avc: denied { lock } for pid=70007 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835986.078:18490): avc: denied { open } for pid=105146 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835986.078:18490): avc: denied { read } for pid=105146 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835986.078:18492): avc: denied { map } for pid=105146 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835022.039:17318): avc: denied { read } for pid=886 comm="sssd" name="resolv.conf" dev="sda1" ino=394967 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834644.311:420): avc: denied { read } for pid=7000 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834644.312:422): avc: denied { map } for pid=7000 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834922.558:15024): avc: denied { lock } for pid=61342 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835022.393:17351): avc: denied { map } for pid=70007 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834922.558:15023): avc: denied { read } for pid=61342 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835022.393:17349): avc: denied { read } for pid=70007 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834922.558:15025): avc: denied { map } for pid=61342 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834644.311:421): avc: denied { lock } for pid=7000 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834922.558:15023): avc: denied { open } for pid=61342 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139956] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed} ----------------------------------------------------------------- time: 00:17:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139956/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139956/ "2023-01-27T16:02:37.730027+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139957] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} ----------------------------------------------------------------- time: 00:27:54 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139957/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139957/ "2023-01-27T16:12:02.242410+0000 mon.a (mon.0) 150 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139960] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:12 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139960/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139960/ SELinux denials found on ubuntu@smithi122.front.sepia.ceph.com: ['type=AVC msg=audit(1674834726.330:423): avc: denied { map } for pid=6974 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834992.697:15025): avc: denied { lock } for pid=61316 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834726.330:422): avc: denied { lock } for pid=6974 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834726.330:421): avc: denied { open } for pid=6974 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834992.350:15003): avc: denied { search } for pid=6884 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674834992.697:15026): avc: denied { map } for pid=61316 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834992.697:15024): avc: denied { read } for pid=61316 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834726.330:421): avc: denied { read } for pid=6974 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834992.697:15024): avc: denied { open } for pid=61316 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139961] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139961/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139961/ SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1674835178.611:14034): avc: denied { search } for pid=3467 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525260 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835179.667:14039): avc: denied { map } for pid=77408 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834924.217:212): avc: denied { map } for pid=4241 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834924.197:210): avc: denied { open } for pid=4241 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834924.202:211): avc: denied { lock } for pid=4241 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834924.197:210): avc: denied { read } for pid=4241 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835179.667:14037): avc: denied { open } for pid=77408 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835179.667:14037): avc: denied { read } for pid=77408 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835179.667:14038): avc: denied { lock } for pid=77408 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139963] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:01 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139963/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=282cb200a68a4040894ef3dedf0e3164 timeout expired in wait_until_healthy [7139966] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139966/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139966/ SELinux denials found on ubuntu@smithi191.front.sepia.ceph.com: ['type=AVC msg=audit(1674834854.808:423): avc: denied { lock } for pid=6997 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834854.808:424): avc: denied { map } for pid=6997 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834854.808:422): avc: denied { read } for pid=6997 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674834854.808:422): avc: denied { open } for pid=6997 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835123.611:15004): avc: denied { search } for pid=6907 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835123.956:15025): avc: denied { read } for pid=61339 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835123.956:15025): avc: denied { open } for pid=61339 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835123.956:15026): avc: denied { lock } for pid=61339 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835123.956:15027): avc: denied { map } for pid=61339 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139967] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:26:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139967/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139967/ "2023-01-27T16:07:03.707535+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139968] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:47 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139968/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139968/ SELinux denials found on ubuntu@smithi087.front.sepia.ceph.com: ['type=AVC msg=audit(1674835354.116:14040): avc: denied { read } for pid=77397 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835354.116:14041): avc: denied { lock } for pid=77397 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835099.536:213): avc: denied { read } for pid=6335 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835099.536:213): avc: denied { open } for pid=6335 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835479.428:16993): avc: denied { read } for pid=94009 comm="rpm" name="Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835099.551:214): avc: denied { lock } for pid=6335 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835479.428:16995): avc: denied { map } for pid=94009 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525268 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835479.069:16962): avc: denied { read } for pid=708 comm="sssd" name="resolv.conf" dev="sda1" ino=525299 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835479.428:16993): avc: denied { open } for pid=94009 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835099.553:215): avc: denied { map } for pid=6335 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835479.428:16994): avc: denied { lock } for pid=94009 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835352.803:14037): avc: denied { search } for pid=6222 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835354.116:14042): avc: denied { map } for pid=77397 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835354.116:14040): avc: denied { open } for pid=77397 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139969] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:21:52 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139969/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139969/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e050ada508ce4f8b91db65ac91c812f9 timeout expired in wait_until_healthy [7139970] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} ----------------------------------------------------------------- time: 00:10:40 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139970/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139970/ SELinux denials found on ubuntu@smithi057.front.sepia.ceph.com: ['type=AVC msg=audit(1674835182.042:218): avc: denied { map } for pid=6331 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835442.907:14044): avc: denied { lock } for pid=77391 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835568.480:17012): avc: denied { lock } for pid=94011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835443.721:14077): avc: denied { open } for pid=77544 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835443.721:14077): avc: denied { read } for pid=77544 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835443.721:14079): avc: denied { map } for pid=77544 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835441.573:14040): avc: denied { search } for pid=6218 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835182.031:216): avc: denied { open } for pid=6331 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835568.480:17011): avc: denied { open } for pid=94011 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835182.031:216): avc: denied { read } for pid=6331 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835568.480:17011): avc: denied { read } for pid=94011 comm="rpm" name="Packages" dev="sda1" ino=525263 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835182.039:217): avc: denied { lock } for pid=6331 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835568.122:16980): avc: denied { read } for pid=689 comm="sssd" name="resolv.conf" dev="sda1" ino=655363 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835568.480:17013): avc: denied { map } for pid=94011 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835443.721:14078): avc: denied { lock } for pid=77544 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835442.907:14045): avc: denied { map } for pid=77391 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835442.906:14043): avc: denied { read } for pid=77391 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835442.906:14043): avc: denied { open } for pid=77391 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139971] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:28:55 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139971/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139971/ "2023-01-27T16:17:19.377344+0000 mon.a (mon.0) 1468 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139972] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/dashboard} ----------------------------------------------------------------- time: 00:20:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139972/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80e8f712fa71429bb6bf19e07fed0805 Test failure: test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) [7139973] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:05:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139973/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bda15d914654eb0ae003e8e0c9349b3 Command failed on smithi161 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7139974] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:20:48 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139974/ "2023-01-27T16:19:22.133971+0000 mon.a (mon.0) 202 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139975] rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:38:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139975/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ac938db298d4077ae604265c329e74b Command failed (workunit test cls/test_cls_rbd.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7139976] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 01:05:30 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139976/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139976/ "2023-01-27T16:18:28.393043+0000 mon.a (mon.0) 422 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139977] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 02:21:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139977/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139977/ "2023-01-27T16:27:15.395400+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139978] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:47 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139978/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139978/ "2023-01-27T16:17:58.574185+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139979] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:29:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139979/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139979/ "2023-01-27T16:25:23.451010+0000 mon.a (mon.0) 390 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139984] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:26 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139984/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139984/ SELinux denials found on ubuntu@smithi073.front.sepia.ceph.com: ['type=AVC msg=audit(1674835730.890:216): avc: denied { map } for pid=6357 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14041): avc: denied { lock } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835983.409:14037): avc: denied { search } for pid=6244 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835730.859:214): avc: denied { read } for pid=6357 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835730.875:215): avc: denied { lock } for pid=6357 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835730.859:214): avc: denied { open } for pid=6357 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14040): avc: denied { open } for pid=77403 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14042): avc: denied { map } for pid=77403 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835984.604:14040): avc: denied { read } for pid=77403 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139986] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 01:18:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139986/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139986/ "2023-01-27T16:25:36.782819+0000 mon.a (mon.0) 316 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139987] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:19:53 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139987/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139987/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b88677b351a048a7bd49c369f75cc709 timeout expired in wait_until_healthy [7139988] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_8} tasks/prometheus} ----------------------------------------------------------------- time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139988/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34ef05828a034e57961480425543ca35 Test failure: test_file_sd_command (tasks.mgr.test_prometheus.TestPrometheus) [7139990] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:15:51 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139990/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139990/ SELinux denials found on ubuntu@smithi067.front.sepia.ceph.com: ['type=AVC msg=audit(1674836106.356:14044): avc: denied { map } for pid=77412 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836233.791:16994): avc: denied { read } for pid=94020 comm="rpm" name="Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836105.557:14036): avc: denied { open } for pid=77405 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836233.791:16995): avc: denied { lock } for pid=94020 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836233.434:16963): avc: denied { read } for pid=737 comm="sssd" name="resolv.conf" dev="sda1" ino=525297 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836105.558:14038): avc: denied { map } for pid=77405 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836106.356:14042): avc: denied { open } for pid=77412 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835842.263:213): avc: denied { open } for pid=6365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836105.558:14037): avc: denied { lock } for pid=77405 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836233.791:16994): avc: denied { open } for pid=94020 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525266 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835842.263:213): avc: denied { read } for pid=6365 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836106.356:14042): avc: denied { read } for pid=77412 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835842.282:215): avc: denied { map } for pid=6365 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835842.275:214): avc: denied { lock } for pid=6365 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836106.356:14043): avc: denied { lock } for pid=77412 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836105.557:14036): avc: denied { read } for pid=77405 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836233.791:16996): avc: denied { map } for pid=94020 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525267 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836105.196:14035): avc: denied { search } for pid=6252 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7139991] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:36:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139991/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139991/ "2023-01-27T16:20:49.597685+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139993] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:18:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139993/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139993/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2a83e63060e4aa69d4f6ca50fac7af0 timeout expired in wait_until_healthy [7139994] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:16:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139994/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139994/ SELinux denials found on ubuntu@smithi122.front.sepia.ceph.com: ['type=AVC msg=audit(1674836008.464:15029): avc: denied { map } for pid=61356 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836008.118:15006): avc: denied { search } for pid=6923 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835737.571:424): avc: denied { map } for pid=7013 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835737.571:422): avc: denied { open } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835737.571:423): avc: denied { lock } for pid=7013 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835737.571:422): avc: denied { read } for pid=7013 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836008.464:15028): avc: denied { lock } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836008.463:15027): avc: denied { open } for pid=61356 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836008.463:15027): avc: denied { read } for pid=61356 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139995] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mon} ----------------------------------------------------------------- time: 01:13:24 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139995/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139995/ SELinux denials found on ubuntu@smithi170.front.sepia.ceph.com: ['type=AVC msg=audit(1674835753.309:419): avc: denied { read } for pid=7014 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836026.403:15003): avc: denied { search } for pid=6924 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674836026.751:15026): avc: denied { map } for pid=61358 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836026.751:15024): avc: denied { read } for pid=61358 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836026.751:15025): avc: denied { lock } for pid=61358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835753.309:421): avc: denied { map } for pid=7014 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836026.751:15024): avc: denied { open } for pid=61358 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835753.309:420): avc: denied { lock } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835753.309:419): avc: denied { open } for pid=7014 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139996] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139996/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139996/ SELinux denials found on ubuntu@smithi133.front.sepia.ceph.com: ['type=AVC msg=audit(1674836133.166:17318): avc: denied { read } for pid=69968 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835970.084:13743): avc: denied { read } for pid=56397 comm="rhsmcertd-worke" name="python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.418:15013): avc: denied { search } for pid=6916 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835969.461:13728): avc: denied { read write } for pid=56397 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15034): avc: denied { open } for pid=61439 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17320): avc: denied { map } for pid=69968 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15034): avc: denied { read } for pid=61439 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13766): avc: denied { open } for pid=56758 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13733): avc: denied { lock } for pid=56569 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13766): avc: denied { read } for pid=56758 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13768): avc: denied { map } for pid=56758 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835973.716:13767): avc: denied { lock } for pid=56758 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15035): avc: denied { lock } for pid=61439 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836132.810:17287): avc: denied { read } for pid=886 comm="sssd" name="resolv.conf" dev="sda1" ino=394924 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13730): avc: denied { getattr } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=77502 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13732): avc: denied { read } for pid=56569 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13734): avc: denied { map } for pid=56569 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17318): avc: denied { open } for pid=69968 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13731): avc: denied { map } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=77502 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13729): avc: denied { lock } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835970.084:13743): avc: denied { open } for pid=56397 comm="rhsmcertd-worke" path="/etc/dnf/modules.d/python36.module" dev="sda1" ino=76230 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.826:13732): avc: denied { open } for pid=56569 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836133.166:17319): avc: denied { lock } for pid=69968 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835969.461:13728): avc: denied { open } for pid=56397 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=73736 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836035.764:15036): avc: denied { map } for pid=61439 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7139997] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:31:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139997/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139997/ "2023-01-27T16:23:01.895698+0000 mgr.x (mgr.4099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7139998] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:36:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139998/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139998/ "2023-01-27T16:31:36.174235+0000 mon.a (mon.0) 1024 : cluster [WRN] Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7139999] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:22:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139999/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139999/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0db10747a8a24f95af9aedda62a6bb8a timeout expired in wait_until_healthy [7140001] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:58 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140001/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140001/ SELinux denials found on ubuntu@smithi042.front.sepia.ceph.com: ['type=AVC msg=audit(1674836080.499:15025): avc: denied { lock } for pid=61369 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835796.416:419): avc: denied { read } for pid=7026 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836080.499:15024): avc: denied { open } for pid=61369 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835796.416:421): avc: denied { map } for pid=7026 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836080.499:15024): avc: denied { read } for pid=61369 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836080.499:15026): avc: denied { map } for pid=61369 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674835796.416:420): avc: denied { lock } for pid=7026 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836080.142:15003): avc: denied { search } for pid=6936 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262168 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674835796.416:419): avc: denied { open } for pid=7026 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7140003] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:15:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140003/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140003/ "2023-01-27T16:23:50.530461+0000 mgr.x (mgr.4116) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140004] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/repair_test} ----------------------------------------------------------------- time: 00:17:55 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140004/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140004/ "2023-01-27T16:21:07.479189+0000 mgr.x (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140005] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:15:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140005/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140005/ "2023-01-27T16:26:37.844435+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140006] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:29:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140006/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140006/ SELinux denials found on ubuntu@smithi174.front.sepia.ceph.com: ['type=AVC msg=audit(1674836343.913:14037): avc: denied { read } for pid=77378 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836344.687:14059): avc: denied { map } for pid=77453 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836344.687:14057): avc: denied { read } for pid=77453 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836090.154:213): avc: denied { lock } for pid=6333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836344.687:14058): avc: denied { lock } for pid=77453 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836090.147:212): avc: denied { read } for pid=6333 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836090.147:212): avc: denied { open } for pid=6333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836343.914:14038): avc: denied { lock } for pid=77378 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836090.161:214): avc: denied { map } for pid=6333 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836344.687:14057): avc: denied { open } for pid=77453 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836343.913:14037): avc: denied { open } for pid=77378 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836343.914:14039): avc: denied { map } for pid=77378 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836342.739:14034): avc: denied { search } for pid=6220 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7140007] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140007/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140007/ "2023-01-27T16:29:40.039817+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140008] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:13:59 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140008/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140008/ "2023-01-27T16:26:51.024910+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140010] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:59:15 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140010/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140010/ "2023-01-27T16:35:36.181625+0000 mon.h (mon.5) 1967 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7140013] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140013/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140013/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf3fb3e83d454e8e917b262ab0bbcec2 timeout expired in wait_until_healthy [7140014] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:21:33 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140014/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140014/ "2023-01-27T16:26:21.694599+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log [7140015] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:14:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140015/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140015/ SELinux denials found on ubuntu@smithi053.front.sepia.ceph.com: ['type=AVC msg=audit(1674836596.453:14040): avc: denied { map } for pid=77391 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836333.054:216): avc: denied { map } for pid=6344 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836333.043:215): avc: denied { lock } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836333.030:214): avc: denied { open } for pid=6344 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836596.091:14037): avc: denied { search } for pid=6231 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674836596.453:14039): avc: denied { lock } for pid=77391 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836596.453:14038): avc: denied { open } for pid=77391 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836596.453:14038): avc: denied { read } for pid=77391 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836333.030:214): avc: denied { read } for pid=6344 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7140016] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140016/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140016/ SELinux denials found on ubuntu@smithi161.front.sepia.ceph.com: ['type=AVC msg=audit(1674836327.961:212): avc: denied { read } for pid=6330 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836705.669:16993): avc: denied { lock } for pid=93987 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=395810 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836581.943:14037): avc: denied { open } for pid=77372 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836705.669:16992): avc: denied { open } for pid=93987 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=395810 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836581.943:14038): avc: denied { lock } for pid=77372 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836327.981:214): avc: denied { map } for pid=6330 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836581.943:14037): avc: denied { read } for pid=77372 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836705.669:16992): avc: denied { read } for pid=93987 comm="rpm" name="Packages" dev="sda1" ino=395810 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836327.961:212): avc: denied { open } for pid=6330 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836705.309:16961): avc: denied { read } for pid=745 comm="sssd" name="resolv.conf" dev="sda1" ino=525276 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836581.943:14039): avc: denied { map } for pid=77372 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836705.669:16994): avc: denied { map } for pid=93987 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=395811 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836327.971:213): avc: denied { lock } for pid=6330 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836581.591:14036): avc: denied { search } for pid=6217 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525260 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1'] [7140017] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140017/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140017/ "2023-01-27T16:36:36.530283+0000 mon.a (mon.0) 97 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log [7140018] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:56 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140018/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140018/ SELinux denials found on ubuntu@smithi115.front.sepia.ceph.com: ['type=AVC msg=audit(1674836182.970:419): avc: denied { read } for pid=7006 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836182.970:419): avc: denied { open } for pid=7006 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836554.254:17307): avc: denied { open } for pid=69871 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836456.739:15026): avc: denied { lock } for pid=61352 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836553.907:17276): avc: denied { read } for pid=873 comm="sssd" name="resolv.conf" dev="sda1" ino=394924 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836182.970:420): avc: denied { lock } for pid=7006 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836182.970:421): avc: denied { map } for pid=7006 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836554.254:17307): avc: denied { read } for pid=69871 comm="rpm" name="Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836456.390:15000): avc: denied { search } for pid=6916 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=262172 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674836456.739:15025): avc: denied { open } for pid=61352 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836554.254:17309): avc: denied { map } for pid=69871 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=394931 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836554.254:17308): avc: denied { lock } for pid=69871 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=394930 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836456.739:15025): avc: denied { read } for pid=61352 comm="rpm" name="Packages" dev="sda1" ino=73848 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674836456.739:15027): avc: denied { map } for pid=61352 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=73849 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1'] [7140020] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:21:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140020/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140020/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e15555dec69499ab99024dbe1a9ace4 timeout expired in wait_until_healthy Dead ================================================================= [7139879] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139879/ Error reimaging machines: 'ssh_keyscan smithi097.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds [7139884] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139884/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139884/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7139924] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:05:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139924/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17f1d9832ffb4d76aa91f9c91a395b00 {'smithi088.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 12, 'changed': True, 'cmd': ['subscription-manager', 'register', '-- activationkey=testnode', '--org=Ceph', '--name=smithi088', '--force'], 'delta': '0:00:00.774678', 'end': '2023-01-27 15:41:48.141999', 'failed_when_result': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager register --activationkey=testnode --org=Ceph --name=smithi088 --force', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 70, 'start': '2023-01-27 15:41:47.367321', 'stderr': 'HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "katello_subscription_facets_host_id"\nDETAIL: Key (host_id)=(407) already exists.', 'stderr_lines': ['HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "katello_subscription_facets_host_id"', 'DETAIL: Key (host_id)=(407) already exists.'], 'stdout': '', 'stdout_lines': []}} [7139992] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:04:51 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139992/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b1194ec07944f278bf32ea8d5557ee3 {'smithi181.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force- confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python- ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'only_upgrade': False, 'package': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi- common'], 'policy_rc_d': None, 'purge': False, 'state': 'absent', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': "'apt-get remove 'librbd1' 'librados2'' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n", 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt- get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock- frontend), is another process using it?\n', 'stderr_lines': ['W: --force- yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 14187 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], 'stdout': '', 'stdout_lines': []}} [7140019] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/workunits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140019/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140019/ Error reimaging machines: 'NoneType' object has no attribute '_fields' Pass ================================================================= [7139711] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:58 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139711/ [7139713] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/progress} time: 00:23:37 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139713/ [7139716] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:09:42 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139716/ [7139718] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:11:38 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139718/ [7139719] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:07:40 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139719/ [7139720] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:30 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139720/ [7139721] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:31:35 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139721/ [7139726] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139726/ [7139729] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:31 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139729/ [7139730] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:46 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139730/ [7139739] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:09:56 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139739/ [7139745] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} time: 00:11:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139745/ [7139747] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 02:52:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139747/ [7139749] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{filestore-xfs} supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:11:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139749/ [7139750] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:14:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139750/ [7139757] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:36 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139757/ [7139758] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 02:26:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139758/ [7139759] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:10:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139759/ [7139760] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:50 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139760/ [7139762] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli} time: 00:17:55 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139762/ [7139764] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:49 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139764/ [7139767] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:58:46 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139767/ [7139771] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139771/ [7139776] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:17:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139776/ [7139778] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139778/ [7139779] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:08:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139779/ [7139780] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:50 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139780/ [7139784] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{ubuntu_latest}} time: 00:30:49 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139784/ [7139788] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:24:35 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139788/ [7139793] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:03:24 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139793/ [7139794] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:37:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139794/ [7139797] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:14 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139797/ [7139798] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:33:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139798/ [7139800] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:10:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139800/ [7139805] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:08 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139805/ [7139809] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:19:24 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139809/ [7139811] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:08:08 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139811/ [7139817] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:18:41 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139817/ [7139823] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:26:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139823/ [7139826] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:24:06 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139826/ [7139828] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139828/ [7139830] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:09:02 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139830/ [7139832] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:21:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139832/ [7139833] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:24:48 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139833/ [7139838] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:15:47 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139838/ [7139842] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139842/ [7139844] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:15:44 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139844/ [7139845] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:08:13 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139845/ [7139848] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:08:34 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139848/ [7139851] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:46 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139851/ [7139852] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:21:39 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139852/ [7139858] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139858/ [7139859] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:01 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139859/ [7139860] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:57 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139860/ [7139862] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:19:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139862/ [7139866] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:00 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139866/ [7139869] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:08:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139869/ [7139870] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:29 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139870/ [7139886] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:31:10 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139886/ [7139890] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:03 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139890/ [7139893] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:19 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139893/ [7139898] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:42 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139898/ [7139905] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:25 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139905/ [7139907] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:14:09 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139907/ [7139914] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:35 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139914/ [7139915] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139915/ [7139916] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139916/ [7139918] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:28:38 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139918/ [7139920] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:08:43 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139920/ [7139929] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:08:11 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139929/ [7139932] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:18:58 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139932/ [7139936] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:23:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139936/ [7139938] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:18:05 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139938/ [7139939] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:08:24 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139939/ [7139940] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:20:08 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139940/ [7139951] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:18 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139951/ [7139952] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:09:20 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139952/ [7139958] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139958/ [7139959] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139959/ [7139962] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:32:21 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139962/ [7139964] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:43 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139964/ [7139965] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:12:42 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139965/ [7139980] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_cephadm_repos} time: 00:06:01 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139980/ [7139981] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:29 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139981/ [7139982] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:33:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139982/ [7139983] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:12:16 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139983/ [7139985] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:04 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139985/ [7139989] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:38:35 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7139989/ [7140000] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 01:09:07 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140000/ [7140002] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:10:28 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140002/ [7140009] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 01:16:45 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140009/ [7140011] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:07:39 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140011/ [7140012] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:01 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140012/ [7140021] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:20:38 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140021/ [7140022] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:13:08 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140022/ [7140023] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:12 info: http://pulpito.ceph.com/yuriw-2023-01-27_01:00:15-rados-quincy-distro-default-smithi/7140023/