2022-06-22T00:04:23.163 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T00:04:23.165 INFO:teuthology.results:subset: '14/16' 2022-06-22T00:04:23.165 INFO:teuthology.results:seed: '7856' 2022-06-22T00:05:34.437 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-22T00:05:34.440 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-22T01:06:39.828 INFO:teuthology.results:Tests finished! gathering results... 2022-06-22T01:06:40.430 INFO:scrape:Found 4 jobs 2022-06-22T01:06:41.953 INFO:scrape:Found 2 distinct failure reasons 2022-06-22T01:06:41.955 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2022-06-22T01:06:41.955 INFO:scrape:2 jobs: ['6890959', '6890961'] 2022-06-22T01:06:41.956 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'tasks/{0-check-counter', 'workunit/db/postgres}}'] 2022-06-22T01:06:41.956 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/5', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'tasks/{0-check-counter', 'workunit/db/postgres}}', 'wsync/yes}'] 2022-06-22T01:06:41.957 INFO:scrape: 2022-06-22T01:06:41.957 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi031.front.sepia.ceph.com: ['type=AVC msg=audit(1655858008.472:21213): avc: denied { rename } for pid=135809 comm="postmaster" name="current_logfiles.tmp" dev="ceph" ino=1099511628782 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { write } for pid=135542 comm="mkdir" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.469:21212): avc: denied { write } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postmaster.opts" dev="ceph" ino=1099511628781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.711:21349): avc: denied { write } for pid=136714 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.532:21233): avc: denied { write } for pid=135844 comm="postmaster" name="1260" dev="ceph" ino=1099511627863 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858187.491:21410): avc: denied { read write } for pid=135812 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858018.041:21236): avc: denied { unlink } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628780 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.711:21348): avc: denied { read } for pid=136714 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858002.969:21078): avc: denied { setattr } for pid=135519 comm="chown" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21210): avc: denied { open } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.638:21343): avc: denied { getattr } for pid=136542 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.630:21342): avc: denied { open } for pid=136542 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858002.949:21071): avc: denied { create } for pid=135515 comm="mkdir" name="postgres" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21210): avc: denied { read } for pid=135803 comm="postmaster" name="postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.022:21294): avc: denied { read write } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.630:21342): avc: denied { read } for pid=136542 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.022:21294): avc: denied { open } for pid=135812 comm="postmaster" path="/tmp/cephfs/postgres/data/pg_logical/replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858024.157:21237): avc: denied { rename } for pid=135812 comm="postmaster" name="replorigin_checkpoint.tmp" dev="ceph" ino=1099511628791 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.531:21232): avc: denied { open } for pid=135844 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.532:21234): avc: denied { getattr } for pid=135844 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.531:21232): avc: denied { read } for pid=135844 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.458:21211): avc: denied { search } for pid=135803 comm="postmaster" name="postgres" dev="ceph" ino=1099511627781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1 trawcon="system_u:system_r:postgresql_db_t:s0"', 'type=AVC msg=audit(1655858002.949:21071): avc: denied { associate } for pid=135515 comm="mkdir" name="postgres" scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=filesystem permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { add_name } for pid=135542 comm="mkdir" name="data" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858188.785:21411): avc: denied { getattr } for pid=136966 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { create } for pid=135542 comm="mkdir" name="data" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21208): avc: denied { search } for pid=135803 comm="postmaster" name="/" dev="ceph" ino=1099511628283 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858128.711:21348): avc: denied { open } for pid=136714 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.402:21209): avc: denied { getattr } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.023:21295): avc: denied { unlink } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858068.577:21299): avc: denied { getattr } for pid=136396 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858005.585:21145): avc: denied { relabelfrom } for pid=135659 comm="restorecon" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858102.724:21344): avc: denied { write } for pid=136542 comm="postmaster" name="1262" dev="ceph" ino=1099511627856 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.717:21350): avc: denied { getattr } for pid=136714 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.402:21208): avc: denied { search } for pid=135803 comm="postmaster" name="postgres" dev="ceph" ino=1099511627781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1 trawcon="system_u:system_r:postgresql_db_t:s0"', 'type=AVC msg=audit(1655858187.491:21410): avc: denied { open } for pid=135812 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1'] 2022-06-22T01:06:41.958 INFO:scrape:2 jobs: ['6890962', '6890960'] 2022-06-22T01:06:41.958 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/5', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/db/postgres}}', 'wsync/no}'] 2022-06-22T01:06:41.959 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/5', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-no-extra-options}', 'tasks/{0-check-counter', 'workunit/db/postgres}}', 'wsync/no}'] 2022-06-22T01:06:41.959 INFO:scrape: 2022-06-22T01:06:41.960 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6890959] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:51:21 info: http://pulpito.ceph.com/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890959/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890959/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6890960] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:34:12 info: http://pulpito.ceph.com/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890960/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890960/ SELinux denials found on ubuntu@smithi101.front.sepia.ceph.com: ['type=AVC msg=audit(1655858082.973:21410): avc: denied { open } for pid=135716 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/2619" dev="ceph" ino=1099511628568 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857890.954:21222): avc: denied { open } for pid=135716 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857890.954:21222): avc: denied { read write } for pid=135716 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857864.480:21041): avc: denied { setattr } for pid=135436 comm="chown" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857869.871:21173): avc: denied { open } for pid=135707 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858082.973:21410): avc: denied { read write } for pid=135716 comm="postmaster" name="2619" dev="ceph" ino=1099511628568 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857869.871:21171): avc: denied { search } for pid=135707 comm="postmaster" name="/" dev="ceph" ino=1099511628283 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857869.939:21177): avc: denied { write } for pid=135707 comm="postmaster" path="/tmp/cephfs/postgres/data/postmaster.opts" dev="ceph" ino=1099511628781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857990.144:21313): avc: denied { getattr } for pid=136619 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857959.581:21306): avc: denied { read } for pid=136453 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857984.748:21312): avc: denied { open } for pid=135717 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857864.461:21034): avc: denied { associate } for pid=135432 comm="mkdir" name="postgres" scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=filesystem permissive=1', 'type=AVC msg=audit(1655857869.871:21173): avc: denied { read } for pid=135707 comm="postmaster" name="postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857869.943:21178): avc: denied { rename } for pid=135714 comm="postmaster" name="current_logfiles.tmp" dev="ceph" ino=1099511628782 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857893.745:21256): avc: denied { write open } for pid=135716 comm="postmaster" path="/tmp/cephfs/postgres/data/pg_logical/replorigin_checkpoint.tmp" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857864.519:21056): avc: denied { write } for pid=135459 comm="mkdir" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857864.519:21056): avc: denied { add_name } for pid=135459 comm="mkdir" name="data" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857959.588:21307): avc: denied { getattr } for pid=136453 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857867.149:21108): avc: denied { relabelfrom } for pid=135568 comm="restorecon" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857953.558:21288): avc: denied { unlink } for pid=135716 comm="postmaster" name="16390" dev="ceph" ino=1099511628785 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857869.871:21172): avc: denied { getattr } for pid=135707 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857864.461:21034): avc: denied { create } for pid=135432 comm="mkdir" name="postgres" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655857930.041:21259): avc: denied { getattr } for pid=136294 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857893.745:21257): avc: denied { read } for pid=135716 comm="postmaster" name="replorigin_checkpoint.tmp" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857959.588:21308): avc: denied { write } for pid=136453 comm="postmaster" name="2601" dev="ceph" ino=1099511628487 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857984.748:21312): avc: denied { read write } for pid=135717 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857893.757:21258): avc: denied { unlink } for pid=135716 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628792 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857869.871:21171): avc: denied { search } for pid=135707 comm="postmaster" name="postgres" dev="ceph" ino=1099511627781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1 trawcon="system_u:system_r:postgresql_db_t:s0"', 'type=AVC msg=audit(1655857893.757:21258): avc: denied { rename } for pid=135716 comm="postmaster" name="replorigin_checkpoint.tmp" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857878.800:21195): avc: denied { unlink } for pid=135716 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628780 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655857959.581:21306): avc: denied { open } for pid=136453 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1'] [6890961] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:45:29 info: http://pulpito.ceph.com/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890961/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890961/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6890962] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/db/postgres}} ----------------------------------------------------------------- time: 00:37:20 info: http://pulpito.ceph.com/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890962/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-06-22_00:04:06-fs:workload-main-distro-default-smithi/6890962/ SELinux denials found on ubuntu@smithi031.front.sepia.ceph.com: ['type=AVC msg=audit(1655858008.472:21213): avc: denied { rename } for pid=135809 comm="postmaster" name="current_logfiles.tmp" dev="ceph" ino=1099511628782 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { write } for pid=135542 comm="mkdir" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.469:21212): avc: denied { write } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postmaster.opts" dev="ceph" ino=1099511628781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.711:21349): avc: denied { write } for pid=136714 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.532:21233): avc: denied { write } for pid=135844 comm="postmaster" name="1260" dev="ceph" ino=1099511627863 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858187.491:21410): avc: denied { read write } for pid=135812 comm="postmaster" name="16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858018.041:21236): avc: denied { unlink } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628780 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.711:21348): avc: denied { read } for pid=136714 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858002.969:21078): avc: denied { setattr } for pid=135519 comm="chown" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21210): avc: denied { open } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.638:21343): avc: denied { getattr } for pid=136542 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.630:21342): avc: denied { open } for pid=136542 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858002.949:21071): avc: denied { create } for pid=135515 comm="mkdir" name="postgres" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21210): avc: denied { read } for pid=135803 comm="postmaster" name="postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.022:21294): avc: denied { read write } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858102.630:21342): avc: denied { read } for pid=136542 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.022:21294): avc: denied { open } for pid=135812 comm="postmaster" path="/tmp/cephfs/postgres/data/pg_logical/replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858024.157:21237): avc: denied { rename } for pid=135812 comm="postmaster" name="replorigin_checkpoint.tmp" dev="ceph" ino=1099511628791 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.531:21232): avc: denied { open } for pid=135844 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.532:21234): avc: denied { getattr } for pid=135844 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.531:21232): avc: denied { read } for pid=135844 comm="postmaster" name="pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.458:21211): avc: denied { search } for pid=135803 comm="postmaster" name="postgres" dev="ceph" ino=1099511627781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1 trawcon="system_u:system_r:postgresql_db_t:s0"', 'type=AVC msg=audit(1655858002.949:21071): avc: denied { associate } for pid=135515 comm="mkdir" name="postgres" scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=filesystem permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { add_name } for pid=135542 comm="mkdir" name="data" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858188.785:21411): avc: denied { getattr } for pid=136966 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858003.009:21093): avc: denied { create } for pid=135542 comm="mkdir" name="data" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858008.402:21208): avc: denied { search } for pid=135803 comm="postmaster" name="/" dev="ceph" ino=1099511628283 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858128.711:21348): avc: denied { open } for pid=136714 comm="postmaster" path="/tmp/cephfs/postgres/data/global/pg_filenode.map" dev="ceph" ino=1099511628113 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.402:21209): avc: denied { getattr } for pid=135803 comm="postmaster" path="/tmp/cephfs/postgres/data/postgresql.conf" dev="ceph" ino=1099511627807 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858038.023:21295): avc: denied { unlink } for pid=135812 comm="postmaster" name="replorigin_checkpoint" dev="ceph" ino=1099511628793 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858068.577:21299): avc: denied { getattr } for pid=136396 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858005.585:21145): avc: denied { relabelfrom } for pid=135659 comm="restorecon" name="postgres" dev="ceph" ino=1099511627781 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:postgresql_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1655858102.724:21344): avc: denied { write } for pid=136542 comm="postmaster" name="1262" dev="ceph" ino=1099511627856 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858128.717:21350): avc: denied { getattr } for pid=136714 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/PG_VERSION" dev="ceph" ino=1099511628683 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1655858008.402:21208): avc: denied { search } for pid=135803 comm="postmaster" name="postgres" dev="ceph" ino=1099511627781 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1 trawcon="system_u:system_r:postgresql_db_t:s0"', 'type=AVC msg=audit(1655858187.491:21410): avc: denied { open } for pid=135812 comm="postmaster" path="/tmp/cephfs/postgres/data/base/12347/16396" dev="ceph" ino=1099511628789 scontext=system_u:system_r:postgresql_t:s0 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1']