You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 12:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 23, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17350995430,14cyclictest29566irq/58-eth022:10:133
17350995430,14cyclictest29566irq/58-eth022:10:123
1732799512,41cyclictest3210-21ssh23:05:141
17339994818,8cyclictest17302-21cyclictest20:35:152
1735099455,25cyclictest29566irq/58-eth000:25:143
1735099453,22cyclictest29566irq/58-eth000:20:153
1733999442,32cyclictest26346irq/40-dwc2_hso20:50:132
1731599422,35cyclictest11708-21ssh23:10:140
1733999414,18cyclictest261rcuc/200:30:132
1733999410,19cyclictest20583-21ssh21:20:142
1732799412,27cyclictest18972-21ssh22:55:121
1731599404,32cyclictest14534-21ssh00:10:130
1732799390,31cyclictest28652-21ssh23:20:121
17339993810,23cyclictest0-21swapper/223:40:142
17339993729,3cyclictest28985-21ssh21:25:152
1732799371,4cyclictest131rcu_preempt23:45:161
1735099362,21cyclictest29566irq/58-eth022:35:153
1733999363,25cyclictest27-21ksoftirqd/223:45:132
17315993623,8cyclictest4341-21apt-get19:30:140
1735099353,15cyclictest29566irq/58-eth023:00:143
17315993519,10cyclictest0-21swapper/021:10:120
1735099335,17cyclictest29566irq/58-eth021:25:143
1733999334,9cyclictest3422-21diskmemload22:30:132
17327993318,10cyclictest629-21Async14:00:481
1735099321,15cyclictest29566irq/58-eth022:40:143
1733999321,18cyclictest26346irq/40-dwc2_hso22:45:132
1732799321,24cyclictest14370-21ssh00:30:151
17315993222,6cyclictest15392-21apt-get20:15:150
1731599322,25cyclictest3422-21diskmemload21:15:140
1731599322,22cyclictest849-21ssh22:25:140
17315993219,9cyclictest26419-21apt-get21:00:140
1735099311,15cyclictest29566irq/58-eth000:35:143
1735099311,15cyclictest29566irq/58-eth000:35:143
1733999311,23cyclictest261rcuc/222:05:132
1732799312,22cyclictest13186-21rm23:30:131
1732799311,5cyclictest131rcu_preempt23:50:131
1732799310,25cyclictest0-21swapper/122:15:141
1732799310,25cyclictest0-21swapper/122:15:131
1735099302,10cyclictest0-21swapper/320:35:153
17350993016,9cyclictest29566irq/58-eth022:15:153
17350993016,9cyclictest29566irq/58-eth022:15:143
1735099301,21cyclictest29566irq/58-eth022:50:153
1733999302,10cyclictest0-21swapper/222:15:142
1733999302,10cyclictest0-21swapper/222:15:132
1732799302,22cyclictest13436-21ssh21:35:171
1731599304,18cyclictest22392-21ssh00:35:140
1731599304,18cyclictest22392-21ssh00:35:130
17315993024,3cyclictest9250-21apt-get19:35:150
17315993024,3cyclictest28644-21apt-get23:20:120
17315993017,8cyclictest10601-21apt-get20:10:130
1735099292,7cyclictest0-21swapper/323:25:193
1733999292,20cyclictest0-21swapper/219:45:142
1732799291,22cyclictest0-21swapper/119:30:151
1732799291,21cyclictest22713-21apt-get23:55:151
1731599292,20cyclictest7780-21ssh22:10:130
1731599292,20cyclictest7780-21ssh22:10:120
1731599292,18cyclictest17698-21ssh22:35:150
1735099283,15cyclictest0-21swapper/322:20:143
1733999282,3cyclictest131rcu_preempt21:35:172
1733999282,16cyclictest26346irq/40-dwc2_hso19:40:142
17339992811,12cyclictest27-21ksoftirqd/200:10:122
1732799285,8cyclictest131rcu_preempt20:05:131
1731599287,5cyclictest151rcuc/023:38:060
17315992821,4cyclictest6019-21apt-get23:45:130
1735099274,12cyclictest29666irq/59-eth023:10:133
1735099272,9cyclictest0-21swapper/321:00:153
1733999276,16cyclictest0-21swapper/221:00:132
1733999274,14cyclictest26346irq/40-dwc2_hso20:15:152
1733999272,20cyclictest0-21swapper/222:00:142
1733999272,18cyclictest0-21swapper/221:05:152
1733999271,14cyclictest4932-21ssh21:30:122
1732799277,15cyclictest22665-21ssh00:15:141
1731599273,7cyclictest12-21ksoftirqd/021:25:150
1731599273,17cyclictest30095-21ssh23:40:140
17315992719,5cyclictest18966-21apt-get22:55:130
1731599271,10cyclictest501-21ssh23:01:460
1735099269,6cyclictest0-21swapper/323:50:513
1735099262,16cyclictest29566irq/58-eth022:55:133
1733999261,3cyclictest0-21swapper/219:25:162
1733999261,18cyclictest15240-21ls23:10:242
1731599265,5cyclictest12-21ksoftirqd/021:49:200
1731599265,4cyclictest6881-21awk23:05:240
1731599262,20cyclictest30789-21ssh00:20:170
17315992618,5cyclictest23292-21apt-get22:00:140
17315992612,10cyclictest4931-21ssh21:30:130
1731599260,11cyclictest24208-21cat20:20:310
1735099253,15cyclictest0-21swapper/323:30:133
1735099251,10cyclictest0-21swapper/300:00:193
1733999252,16cyclictest0-21swapper/219:35:052
1733999251,16cyclictest11434-21ls23:10:012
17327992517,4cyclictest0-21swapper/100:00:171
17327992514,4cyclictest21-21ksoftirqd/119:35:141
1732799251,17cyclictest295166ssh22:05:251
1731599254,5cyclictest12-21ksoftirqd/023:56:350
1731599253,4cyclictest12-21ksoftirqd/000:15:220
1731599252,17cyclictest6604-21ssh00:05:130
1735099242,12cyclictest29566irq/58-eth000:05:123
1733999248,7cyclictest27-21ksoftirqd/220:10:262
1733999248,14cyclictest3471-21awk00:00:362
1733999248,12cyclictest27-21ksoftirqd/223:50:132
1733999243,16cyclictest26346irq/40-dwc2_hso23:30:142
1733999241,9cyclictest6641-21hwlatdetect21:50:162
17339992413,9cyclictest24597-21ssh22:56:442
17339992411,10cyclictest22150-21fschecks_count21:40:172
17339992410,9cyclictest0-21swapper/222:10:132
17339992410,9cyclictest0-21swapper/222:10:122
17339992410,11cyclictest22035-21sh22:35:322
1733999240,19cyclictest3849-21cpuspeed21:10:132
1732799243,14cyclictest0-21swapper/121:00:141
1732799241,15cyclictest9988-21ssh21:50:251
1731599247,4cyclictest0-21swapper/021:21:130
1731599246,5cyclictest0-21swapper/000:27:280
1731599243,4cyclictest12-21ksoftirqd/022:42:160
1731599242,5cyclictest12-21ksoftirqd/021:50:250
1731599241,9cyclictest28667-21mailstats20:25:310
17315992417,4cyclictest5772-21apt-get20:05:140
17350992310,3cyclictest0-21swapper/323:56:063
1733999238,12cyclictest0-21swapper/220:45:152
1733999237,10cyclictest27-21ksoftirqd/221:45:142
1733999231,17cyclictest24893-21ssh22:20:172
1732799231,3cyclictest131rcu_preempt00:10:141
1732799231,18cyclictest30300-21grep23:40:161
1732799231,18cyclictest15383-21find20:15:141
1731599237,4cyclictest0-21swapper/023:50:040
1731599236,4cyclictest0-21swapper/000:33:280
1731599236,12cyclictest0-21swapper/022:50:150
1731599235,13cyclictest0-21swapper/021:55:130
1731599234,4cyclictest12-21ksoftirqd/023:17:530
1731599233,5cyclictest18822-21ssh23:31:390
1731599232,8cyclictest4760-21sh23:25:150
17315992315,6cyclictest13418-21apt-get21:35:160
1731599231,18cyclictest31056-21sh00:00:170
1735099223,11cyclictest29566irq/58-eth023:15:143
1735099222,12cyclictest29566irq/58-eth023:45:143
1733999223,8cyclictest27-21ksoftirqd/200:25:152
1733999222,10cyclictest0-21swapper/220:05:142
1733999221,9cyclictest807-21/usr/sbin/munin22:25:232
1732799223,4cyclictest21-21ksoftirqd/122:45:121
1732799221,18cyclictest2475-21ls20:35:181
1732799221,17cyclictest13993-21latency_hist00:30:011
1732799221,17cyclictest11693-21apt-get23:10:131
1732799221,16cyclictest11935-21apt-get20:45:131
1731599226,4cyclictest0-21swapper/022:47:360
1731599225,14cyclictest0-21swapper/022:30:130
1731599224,4cyclictest12-21ksoftirqd/019:25:200
1731599223,6cyclictest25678-21ssh21:40:290
1731599223,12cyclictest0-21swapper/019:55:170
1731599223,11cyclictest662-21ntpd21:05:140
1731599222,4cyclictest12-21ksoftirqd/022:15:190
1731599222,4cyclictest12-21ksoftirqd/022:15:180
1731599221,8cyclictest0-21swapper/022:05:120
1731599221,14cyclictest26864-21cron19:20:010
1735099213,7cyclictest0-21swapper/320:10:023
1735099213,4cyclictest0-21swapper/321:40:013
1735099213,12cyclictest0-21swapper/321:45:153
1733999219,5cyclictest18301-21cat21:55:252
1733999212,14cyclictest0-21swapper/200:35:152
1733999212,14cyclictest0-21swapper/200:35:142
1733999211,9cyclictest2849-21idleruntime-cro23:05:012
1733999211,8cyclictest2638866ssh22:40:162
1733999211,11cyclictest28999-21ps19:20:202
1733999211,10cyclictest25425-21grep23:35:272
1733999210,18cyclictest6577-21find00:05:102
17327992113,6cyclictest29691-21cron23:40:011
1731599217,4cyclictest0-21swapper/022:21:100
1731599217,11cyclictest12105-21fschecks_time20:45:170
1735099207,9cyclictest0-21swapper/321:15:153
1735099203,4cyclictest0-21swapper/319:15:013
1735099202,8cyclictest0-21swapper/323:40:143
1735099201,12cyclictest0-21swapper/320:15:003
1733999209,4cyclictest24654-21sh23:15:312
1733999208,6cyclictest0-21swapper/219:50:132
1733999208,3cyclictest0-21swapper/223:28:032
1733999205,7cyclictest713-21munin-plugin-st20:00:022
1733999205,12cyclictest0-21swapper/219:15:202
1733999201,4cyclictest0-21swapper/220:30:132
1733999201,16cyclictest25272-21grep20:25:202
1733999201,15cyclictest932-21snmpd20:55:152
1733999201,15cyclictest7423-21sed20:40:182
1733999201,15cyclictest31010-21fschecks_time00:20:192
1732799203,7cyclictest21-21ksoftirqd/119:40:161
1732799202,15cyclictest27323-21ssh23:00:131
1732799202,14cyclictest4756-21ssh23:25:151
17327992012,6cyclictest26843-21sshd19:16:431
17327992012,5cyclictest0-21swapper/119:45:271
1732799201,15cyclictest26985-21ssh22:00:271
1732799201,15cyclictest24760-21find22:20:121
1732799200,14cyclictest25931-21missed_timers00:35:251
1732799200,14cyclictest25931-21missed_timers00:35:241
1731599208,9cyclictest2417-21tune2fs20:35:180
1731599203,12cyclictest0-21swapper/020:55:140
1731599202,14cyclictest27291-21fschecks_time19:20:160
1731599201,9cyclictest30081-21grep20:30:160
17315992011,5cyclictest23719-21apt-get19:50:140
1735099192,11cyclictest0-21swapper/319:55:173
1735099192,10cyclictest29566irq/58-eth022:30:143
1735099191,15cyclictest29566irq/58-eth021:40:153
1735099191,13cyclictest0-21swapper/321:30:153
1735099191,13cyclictest0-21swapper/319:50:023
1735099191,12cyclictest0-21swapper/300:10:163
1735099191,11cyclictest0-21swapper/319:40:183
1733999196,10cyclictest17405-21latency_hist19:10:022
1733999196,10cyclictest17405-21latency_hist19:10:012
1733999192,8cyclictest8208-21unixbench_singl19:30:302
1733999191,9cyclictest17964-21ps19:10:202
1733999191,8cyclictest30512-21kthreadcore23:20:222
1733999191,15cyclictest22341-21kthreadcore20:20:232
1733999191,14cyclictest23345-21irqstats00:15:192
1733999191,13cyclictest22731-21ssh23:55:162
1733999191,10cyclictest12706-21ps21:15:202
17339991910,7cyclictest0-21swapper/220:00:262
1733999190,11cyclictest12037-21kthreadcore22:50:212
1732799193,13cyclictest12160-21ssh21:15:151
1732799191,9cyclictest3753-21/usr/sbin/munin21:10:161
1732799191,9cyclictest18317-21kthreadcore20:50:211
17327991912,4cyclictest0-21swapper/122:25:281
1732799191,15cyclictest6598-21ssh00:05:111
1732799191,15cyclictest19681-21users21:55:331
17327991911,5cyclictest0-21swapper/119:25:171
1732799191,12cyclictest12990-21ssh22:30:261
1732799190,14cyclictest0-21swapper/120:10:151
1731599197,9cyclictest7736-21uname20:40:190
1735099188,6cyclictest29566irq/58-eth023:20:113
1735099188,3cyclictest29666irq/59-eth021:10:133
1735099182,10cyclictest0-21swapper/320:45:143
1735099182,10cyclictest0-21swapper/320:30:143
1735099182,10cyclictest0-21swapper/320:20:173
1735099182,10cyclictest0-21swapper/320:00:153
1735099181,10cyclictest0-21swapper/319:30:183
17350991810,6cyclictest0-21swapper/323:09:103
1732799189,6cyclictest9025-21kthreadcore22:10:201
1732799189,6cyclictest9025-21kthreadcore22:10:191
1732799188,7cyclictest10976-21grep20:40:301
1732799184,3cyclictest131rcu_preempt22:35:341
1732799183,3cyclictest131rcu_preempt22:40:121
1732799183,11cyclictest6077-21latency_hist00:25:031
1732799182,13cyclictest21848-21ssh21:40:041
1732799182,11cyclictest0-21swapper/120:30:151
1732799181,14cyclictest31114-21kthreadcore21:25:221
1732799181,14cyclictest20173-21ssh23:15:141
1732799181,13cyclictest10690-21apt-get22:50:151
1732799180,12cyclictest28543-21find19:55:131
1731599186,8cyclictest20922-21ntp_states19:10:270
1731599183,11cyclictest20788-21kthreadcore19:45:210
1731599182,11cyclictest842-21munin-node20:50:140
1731599181,10cyclictest0-21swapper/020:00:250
1735099179,6cyclictest0-21swapper/321:20:493
1735099179,6cyclictest0-21swapper/320:51:143
1735099179,6cyclictest0-21swapper/300:30:063
1735099179,5cyclictest29566irq/58-eth023:35:133
1735099179,5cyclictest29566irq/58-eth022:45:133
1735099179,5cyclictest29566irq/58-eth022:05:123
1735099179,3cyclictest0-21swapper/322:25:083
1735099173,8cyclictest0-21swapper/322:00:133
1735099172,10cyclictest0-21swapper/319:35:153
1735099172,10cyclictest0-21swapper/300:15:153
17350991711,4cyclictest0-21swapper/321:59:443
1732799179,6cyclictest20569-21apt-get21:20:141
1732799178,6cyclictest24812-21cron20:25:021
1732799175,9cyclictest25552-21grep20:25:211
1732799173,11cyclictest29365-21kthreadcore19:20:221
1732799171,7cyclictest26178-21run-parts21:00:001
1732799171,14cyclictest4896-21/usr/sbin/munin21:30:131
1732799171,13cyclictest22212-21ls19:15:021
1735099169,3cyclictest0-21swapper/321:51:483
1735099162,10cyclictest0-21swapper/321:05:143
1732799161,12cyclictest30357-21/usr/sbin/munin21:45:131
1732799161,11cyclictest2298-21ntp_states21:05:261
1732799161,11cyclictest1417-21irqcore20:00:201
1731599160,11cyclictest17816-21timerandwakeup19:40:300
1735099152,10cyclictest0-21swapper/320:15:173
1731599143,8cyclictest17409-21latency_hist19:10:020
1731599143,8cyclictest17409-21latency_hist19:10:010
1735099132,9cyclictest0-21swapper/320:40:153
1735099132,9cyclictest0-21swapper/320:25:173
1735099132,4cyclictest0-21swapper/319:25:143
1732799131,8cyclictest17328-21latency_hist19:10:011
1732799131,8cyclictest17328-21latency_hist19:10:011
1735099122,6cyclictest0-21swapper/319:50:143
1735099122,6cyclictest0-21swapper/319:15:143
1735099122,5cyclictest0-21swapper/320:55:133
1735099121,9cyclictest0-21swapper/319:20:143
1735099101,6cyclictest0-21swapper/319:10:023
1735099101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional