### List the top n busiest threads with detailed stack trace `thread -n 3`{{execute T2}} ```bash $ thread -n 3 "C1 CompilerThread0" [Internal] cpuUsage=1.63% deltaTime=3ms time=1170ms "arthas-command-execute" Id=23 cpuUsage=0.11% deltaTime=0ms time=401ms RUNNABLE at java.management@11.0.7/sun.management.ThreadImpl.dumpThreads0(Native Method) at java.management@11.0.7/sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:466) at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:199) at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:122) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108) at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385) at java.base@11.0.7/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.7/java.lang.Thread.run(Thread.java:834) "VM Periodic Task Thread" [Internal] cpuUsage=0.07% deltaTime=0ms time=584ms ``` * Without thread ID, including `[Internal]` means JVM internal thread, refer to the introduction of `dashboard` command. * `cpuUsage` is the CPU usage of the thread during the sampling interval, consistent with the data of the `dashboard` command. * `deltaTime` is the incremental CPU time of the thread during the sampling interval. If it is less than 1ms, it will be rounded and displayed as 0ms. * `time` The total CPU time of thread. **Note:** The thread stack is acquired at the end of the second sampling, which does not indicate that the thread is processing the same task during the sampling interval. It is recommended that the interval time should not be too long. The larger the interval time, the more inaccurate. You can try to specify different intervals according to the specific situation and observe the output results. #### List first page threads' info when no options provided By default, they are arranged in descending order of CPU increment time, and only the first page of data is displayed. `thread`{{execute T2}} ```bash $ thread Threads Total: 33, NEW: 0, RUNNABLE: 9, BLOCKED: 0, WAITING: 3, TIMED_WAITING: 4, TERMINATED: 0, Internal threads: 17 ID NAME GROUP PRIORITY STATE %CPU DELTA_TIME TIME INTERRUPT DAEMON -1 C2 CompilerThread0 - -1 - 5.06 0.010 0:0.973 false true -1 C1 CompilerThread0 - -1 - 0.95 0.001 0:0.603 false true 23 arthas-command-execute system 5 RUNNABLE 0.17 0.000 0:0.226 false true -1 VM Periodic Task Thread - -1 - 0.05 0.000 0:0.094 false true -1 Sweeper thread - -1 - 0.04 0.000 0:0.011 false true -1 G1 Young RemSet Sampling - -1 - 0.02 0.000 0:0.025 false true 12 Attach Listener system 9 RUNNABLE 0.0 0.000 0:0.022 false true 11 Common-Cleaner InnocuousThrea 8 TIMED_WAI 0.0 0.000 0:0.000 false true 3 Finalizer system 8 WAITING 0.0 0.000 0:0.000 false true 2 Reference Handler system 10 RUNNABLE 0.0 0.000 0:0.000 false true 4 Signal Dispatcher system 9 RUNNABLE 0.0 0.000 0:0.000 false true 15 arthas-NettyHttpTelnetBootstra system 5 RUNNABLE 0.0 0.000 0:0.029 false true 22 arthas-NettyHttpTelnetBootstra system 5 RUNNABLE 0.0 0.000 0:0.196 false true 24 arthas-NettyHttpTelnetBootstra system 5 RUNNABLE 0.0 0.000 0:0.038 false true 16 arthas-NettyWebsocketTtyBootst system 5 RUNNABLE 0.0 0.000 0:0.001 false true 17 arthas-NettyWebsocketTtyBootst system 5 RUNNABLE 0.0 0.000 0:0.001 false true ``` ### thread id, show the running stack for the target thread View the stack of thread ID 16: `thread 16`{{execute T2}} ```bash $ thread 1 "main" Id=1 WAITING on java.util.concurrent.CountDownLatch$Sync@29fafb28 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.CountDownLatch$Sync@29fafb28 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ``` ### thread -b, locate the thread bocking the others In some occasions, we experience the whole application is stuck because there’s one particular thread hold one lock that other threads are relying on. To diagnose such an issue, Arthas provides thread -b to find the problematic thread in one single command. `thread -b`{{execute T2}} ```bash $ thread -b "http-bio-8080-exec-4" Id=27 TIMED_WAITING at java.lang.Thread.sleep(Native Method) at test.arthas.TestThreadBlocking.doGet(TestThreadBlocking.java:22) - locked java.lang.Object@725be470 <---- but blocks 4 other threads! at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at test.filter.TestDurexFilter.doFilter(TestDurexFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at com.taobao.tomcat.valves.ContextLoadFilterValve$FilterChainAdapter.doFilter(ContextLoadFilterValve.java:191) at com.taobao.eagleeye.EagleEyeFilter.doFilter(EagleEyeFilter.java:81) at com.taobao.tomcat.valves.ContextLoadFilterValve.invoke(ContextLoadFilterValve.java:150) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:429) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) - locked org.apache.tomcat.util.net.SocketWrapper@7127ee12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@31a6493e ``` **Note**: By now Arthas only supports to locate the thread blocked by synchronzied, while `java.util.concurrent.Lock` is not supported yet. ### thread -i, specify the sampling interval * `thread -i 1000`: Count the thread cpu time of the last 1000ms. `thread -i 1000`{{execute T2}} * `thread -n 3 -i 1000`: List the 3 busiest thread stacks in 1000ms. `thread -n 3 -i 1000`{{execute T2}} ```bash $ thread -n 3 -i 1000 "as-command-execute-daemon" Id=4759 cpuUsage=23% RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440) at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:133) at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:79) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:96) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:27) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:125) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:122) at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:332) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:756) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@546aeec1 ... ``` ### thread –state , view the special state theads `thread --state WAITING`{{execute T2}} ```bash [arthas@28114]$ thread --state WAITING Threads Total: 16, NEW: 0, RUNNABLE: 9, BLOCKED: 0, WAITING: 3, TIMED_WAITING: 4, TERMINATED: 0 ID NAME GROUP PRIORITY STATE %CPU DELTA_TIME TIME INTERRUPTE DAEMON 3 Finalizer system 8 WAITING 0.0 0.000 0:0.000 false true 20 arthas-UserStat system 9 WAITING 0.0 0.000 0:0.001 false true 14 arthas-timer system 9 WAITING 0.0 0.000 0:0.000 false true ```