[BUG] ActionPrivileges initialization can take a long time in clusters with a large number of roles with repeated index patterns #5464
Description
What is the bug?
With Optimized Privilege Evaluation, much of the work of privilege evaluation is now done upfront by creating optimized data structures on node boostrap or if an index is created/deleted (change to cluster state)
An issue is observed, where the initialization of the ActionPrivileges takes a long time in clusters with a large number of roles that have the same repeated index patterns across role definitions.
Below is a snapshot of the hot threads output:
69.6% (348.1ms out of 500ms) cpu usage by thread 'opensearch[f97f9501b3e26a987f62a225d50caa46][generic][T#8]'
5/10 snapshots sharing following 30 elements
org.opensearch.security.support.WildcardMatcher$SimpleMatcher.test(WildcardMatcher.java:487)
org.opensearch.security.support.WildcardMatcher$SimpleMatcher.test(WildcardMatcher.java:461)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.lambda$test$0(WildcardMatcher.java:530)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner$$Lambda/0x000000020343a9c0.test(Unknown Source)
java.base@21.0.6/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
java.base@21.0.6/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034)
java.base@21.0.6/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
java.base@21.0.6/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
java.base@21.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
java.base@21.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
java.base@21.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.base@21.0.6/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:530)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:517)
org.opensearch.security.support.WildcardMatcher$3$1.init(WildcardMatcher.java:324)
org.opensearch.security.support.WildcardMatcher$3$1.hasNext(WildcardMatcher.java:303)
org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1014)
org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247)
org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges$$Lambda/0x00000002035066e0.run(Unknown Source)
java.base@21.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
java.base@21.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:964)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.6/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 23 elements
java.base@21.0.6/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:526)
java.base@21.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
java.base@21.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
java.base@21.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.base@21.0.6/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:530)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:517)
org.opensearch.security.support.WildcardMatcher$3$1.init(WildcardMatcher.java:324)
org.opensearch.security.support.WildcardMatcher$3$1.hasNext(WildcardMatcher.java:303)
org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1014)
org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247)
org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges$$Lambda/0x00000002035066e0.run(Unknown Source)
java.base@21.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
java.base@21.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:964)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.6/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 30 elements
org.opensearch.security.support.WildcardMatcher$SimpleMatcher.test(WildcardMatcher.java:483)
org.opensearch.security.support.WildcardMatcher$SimpleMatcher.test(WildcardMatcher.java:461)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.lambda$test$0(WildcardMatcher.java:530)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner$$Lambda/0x000000020343a9c0.test(Unknown Source)
java.base@21.0.6/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
java.base@21.0.6/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034)
java.base@21.0.6/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
java.base@21.0.6/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
java.base@21.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
java.base@21.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
java.base@21.0.6/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
java.base@21.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.base@21.0.6/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:530)
org.opensearch.security.support.WildcardMatcher$MatcherCombiner.test(WildcardMatcher.java:517)
org.opensearch.security.support.WildcardMatcher$3$1.init(WildcardMatcher.java:324)
org.opensearch.security.support.WildcardMatcher$3$1.hasNext(WildcardMatcher.java:303)
org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1014)
org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247)
org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68)
org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges$$Lambda/0x00000002035066e0.run(Unknown Source)
java.base@21.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
java.base@21.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:964)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.6/java.lang.Thread.run(Thread.java:1583)
Another issue is that updates to ActionPrivileges are performed asynchronously(when cluster state changes) by creating a new thread from the generic threadpool. It may make sense to create a dedicated threadpool for these tasks to not take threads away from the generic pool for re-computing the ActionPrivileges.