You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 08:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Jun 02, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2540099670,31cyclictest17384-21ssh23:20:121
2539199579,5cyclictest131rcu_preempt23:45:140
2541699513,29cyclictest35966irq/58-eth022:00:123
25408995139,9cyclictest11203-21diskmemload21:55:142
2539199511,37cyclictest24608-21ssh22:30:130
2539199481,41cyclictest11595-21apt-get21:10:140
25400994640,3cyclictest21-21ksoftirqd/122:00:121
2541699452,26cyclictest35966irq/58-eth000:00:123
2541699433,29cyclictest35966irq/58-eth021:50:133
2540099432,37cyclictest31848-21ssh23:10:141
2539199431,34cyclictest27907-21sh00:20:130
2539199422,34cyclictest13881-21apt-get23:00:130
2539199421,37cyclictest9079-21apt-get21:45:150
2539199421,33cyclictest5847-21ssh21:25:130
2541699411,22cyclictest35966irq/58-eth020:10:133
25416994111,20cyclictest35966irq/58-eth022:25:143
2540099413,33cyclictest18931-21ssh00:15:131
2540099411,35cyclictest29402-21rm21:20:121
2539199413,32cyclictest30176-21ssh22:15:140
2539199413,29cyclictest15323-21ssh23:55:140
2539199413,29cyclictest15323-21ssh23:55:140
2539199411,35cyclictest0-21swapper/019:20:120
25408994013,19cyclictest25378-21cyclictest23:00:122
2539199401,35cyclictest24623-21apt-get20:50:120
2540099392,34cyclictest9083-21ssh21:45:141
2540099391,33cyclictest13899-21ssh23:00:121
2539199391,32cyclictest18930-21apt-get00:15:150
2539199391,32cyclictest0-21swapper/023:05:130
2539199390,34cyclictest25560-21apt-get19:10:130
2539199382,33cyclictest11831-21apt-get23:35:140
2539199381,25cyclictest25502-21sh21:53:450
2539199380,35cyclictest17373-21apt-get23:20:120
2539199380,31cyclictest21172-21apt-get22:10:120
2540899372,31cyclictest4859-21ntp_states22:35:262
2540099374,28cyclictest21184-21ssh22:10:111
2540099372,23cyclictest1179-21ssh22:35:131
2539199371,29cyclictest0-21swapper/019:50:130
2539199370,31cyclictest22024-21apt-get19:40:130
2539199370,28cyclictest137011cat00:30:140
2541699361,19cyclictest35966irq/58-eth023:40:113
2540099361,27cyclictest0-21swapper/120:35:141
2539199363,28cyclictest0-21swapper/021:40:120
2539199361,31cyclictest19252-21ssh22:45:140
2539199361,29cyclictest0-21swapper/020:30:120
2539199360,21cyclictest0-21swapper/022:41:030
2540899353,19cyclictest15677-21ssh22:25:152
2539199351,31cyclictest10118-21apt-get20:35:140
2539199351,30cyclictest0-21swapper/021:05:140
2539199351,29cyclictest12385-21apt-get19:30:120
2539199351,29cyclictest0-21swapper/022:55:130
2539199350,22cyclictest6577-21ssh00:06:290
2540899346,15cyclictest27346irq/40-dwc2_hso20:20:132
2540899343,24cyclictest27-21ksoftirqd/200:00:122
25408993417,14cyclictest14951-21kthreadcore23:35:222
2539199341,31cyclictest2775-21/usr/sbin/munin23:30:300
2539199341,29cyclictest0-21swapper/000:35:130
2539199341,28cyclictest23249-21apt-get20:15:130
2539199341,21cyclictest27459-21ssh23:42:400
2539199341,20cyclictest7585-21/usr/sbin/munin19:25:220
2541699338,12cyclictest35966irq/58-eth023:00:123
2539199331,27cyclictest26888-21apt-get21:55:120
2539199331,20cyclictest1708-21ssh22:51:400
2539199331,19cyclictest0-21swapper/023:29:360
2539199330,31cyclictest73-21kswapd022:08:580
2541699328,20cyclictest35966irq/58-eth023:15:143
2541699321,19cyclictest35966irq/58-eth019:35:143
2539199324,23cyclictest4572-21apt-get00:25:140
2539199321,29cyclictest21424-21ssh22:26:400
2539199321,29cyclictest17122-21ntp_states20:05:250
2539199321,27cyclictest32209-21cut19:15:190
2539199321,27cyclictest31838-21apt-get23:10:130
2539199321,27cyclictest19502-21sshd20:42:250
2539199321,27cyclictest17314-21diskstats19:35:160
2539199321,25cyclictest0-21swapper/020:45:130
2539199321,20cyclictest15738-21ssh23:18:300
2539199321,18cyclictest6747-21ssh22:36:230
25391993211,17cyclictest10014-21cut20:00:210
25391993211,17cyclictest10014-21cut20:00:200
2539199320,28cyclictest6679-21find22:20:130
2539199320,28cyclictest24249-21apt-get00:00:120
2539199320,19cyclictest11658-21ssh22:04:430
2541699312,19cyclictest35966irq/58-eth022:35:133
2541699311,20cyclictest35966irq/58-eth023:30:133
2541699311,18cyclictest35966irq/58-eth022:10:113
2540899314,14cyclictest27346irq/40-dwc2_hso19:10:122
25391993118,10cyclictest0-21swapper/000:10:140
2539199311,27cyclictest25390-21ls19:10:010
2539199311,24cyclictest0-21swapper/019:55:120
25416993017,7cyclictest35966irq/58-eth021:15:173
2540899308,14cyclictest13625-21sh00:30:132
2539199301,25cyclictest25631-21kthreadcore21:35:210
2539199301,24cyclictest28059-21apt-get20:20:130
2539199300,27cyclictest20744-21ssh21:31:550
2539199300,27cyclictest11832-21ssh23:51:210
25416992913,12cyclictest35966irq/58-eth021:30:123
2540899294,18cyclictest27346irq/40-dwc2_hso19:30:112
2540899291,18cyclictest930-21snmpd20:35:122
2540899291,17cyclictest3363-21ssh22:00:132
2540099293,19cyclictest4576-21ssh00:25:131
2540099291,21cyclictest11858-21ssh23:35:141
2539199291,26cyclictest30320-21ntp_states19:45:270
2539199291,25cyclictest526-21grep20:25:130
2539199291,25cyclictest24243-21ntp_states21:15:300
2539199291,25cyclictest2184-21ssh21:20:590
2539199291,25cyclictest18544-21df20:10:140
2539199291,25cyclictest1825-21/usr/sbin/munin21:00:150
2539199290,25cyclictest29447-21/usr/sbin/munin20:55:150
2540899283,19cyclictest27346irq/40-dwc2_hso22:10:122
2540899282,18cyclictest27346irq/40-dwc2_hso00:25:132
2540099271,20cyclictest13623-21ssh00:30:131
2541699263,15cyclictest0-21swapper/323:48:263
2541699261,18cyclictest0-21swapper/323:08:273
2541699261,17cyclictest0-21swapper/323:55:143
2541699261,17cyclictest0-21swapper/323:55:143
2540899263,19cyclictest22895-21ssh23:05:152
2540899261,9cyclictest11203-21diskmemload23:28:202
2540899261,8cyclictest2478466ssh21:52:532
2540899261,11cyclictest406-21memory21:20:232
2540099261,20cyclictest20556-21ssh21:15:171
2541699252,17cyclictest0-21swapper/323:28:263
2540899252,18cyclictest9265-21ssh23:32:262
2540899251,7cyclictest1815966rm00:14:352
2540899251,6cyclictest2345166ssh23:59:322
2540899251,6cyclictest2345166ssh23:59:322
25408992513,9cyclictest21965-21ssh21:33:242
2540899251,17cyclictest32193-21kthreadcore22:15:232
2540099257,13cyclictest21-21ksoftirqd/120:50:121
25400992512,10cyclictest13024-21cat23:15:311
2541699242,12cyclictest0-21swapper/320:15:133
25416992414,5cyclictest0-21swapper/300:25:143
2541699241,12cyclictest35966irq/58-eth021:25:133
2540899242,6cyclictest0-21swapper/223:47:262
2540899242,17cyclictest0-21swapper/220:32:512
2540899242,16cyclictest0-21swapper/222:33:262
2540899241,7cyclictest2874366ssh00:36:552
2540899241,6cyclictest686066ssh21:42:462
2540899241,5cyclictest29519-21ls20:55:172
25408992413,9cyclictest16835-21ssh21:49:032
2541699232,13cyclictest0-21swapper/322:15:143
2541699232,12cyclictest35966irq/58-eth020:00:133
2541699232,12cyclictest35966irq/58-eth020:00:133
2540899238,8cyclictest31806-21switchtime00:20:282
2540899237,8cyclictest27-21ksoftirqd/200:15:152
2540899231,7cyclictest2363866ssh21:35:112
2540899231,7cyclictest1913766ssh22:45:072
2540899231,6cyclictest2409966ssh23:22:482
2540899231,6cyclictest1277366ssh21:28:042
25408992312,9cyclictest42-21kauditd20:00:012
25408992311,9cyclictest22206-21cut19:40:162
2540899231,19cyclictest1274-21ls19:15:212
2540899231,13cyclictest7190-21ssh23:50:202
25408992310,5cyclictest12394-21kthreadcore22:40:202
2540099239,10cyclictest25378-21cyclictest23:28:201
2540099232,14cyclictest630-21Async09:36:311
25400992311,9cyclictest2943-21cron22:00:011
2540099231,16cyclictest22894-21ssh23:05:151
2541699221,15cyclictest0-21swapper/320:20:123
2540899229,10cyclictest4900-21apt-get22:55:122
2540899228,11cyclictest9063-21ssh23:15:202
2540899227,7cyclictest23637-21kthreadcore21:15:252
2540899221,7cyclictest1282066ssh22:22:022
25408992211,8cyclictest11203-21diskmemload00:05:042
2540899221,12cyclictest1909366ssh22:07:562
2540899221,11cyclictest20908-21rm23:40:132
2540899220,6cyclictest80466ssh22:50:362
2540099221,8cyclictest14800-21cat22:40:301
2541699213,14cyclictest35966irq/58-eth000:05:123
2541699213,13cyclictest0-21swapper/323:54:253
2541699213,12cyclictest0-21swapper/322:30:153
2541699213,10cyclictest35966irq/58-eth021:55:123
2541699212,13cyclictest0-21swapper/322:41:263
25416992110,7cyclictest35966irq/58-eth023:35:143
2540899218,6cyclictest2723-21/usr/sbin/munin19:50:262
2540899212,14cyclictest844-21munin-node21:10:142
2540899212,11cyclictest3093-21awk23:10:262
2540899211,16cyclictest930-21snmpd19:35:142
2540899211,16cyclictest7551-21grep19:25:022
2540899211,10cyclictest23326-21grep20:45:282
2540099212,3cyclictest0-21swapper/122:30:171
2540099212,14cyclictest17542-21sed19:35:181
25400992113,6cyclictest15990-21ssh22:05:261
2540099211,14cyclictest11599-21ssh21:10:131
2540099211,12cyclictest8687-21latency_hist21:45:011
2541699202,13cyclictest35966irq/58-eth022:45:143
25416992012,5cyclictest35966irq/58-eth021:40:123
25416992010,8cyclictest0-21swapper/323:20:163
2540899202,9cyclictest15930-21kthreadcore20:05:202
2540899201,9cyclictest20101-21kthreadcore20:10:202
2540899201,16cyclictest1285-21kthreadcore20:25:172
2540899201,15cyclictest18836-21grep20:40:302
2540899201,11cyclictest27956-21kthreadcore19:45:212
2540899201,11cyclictest26457-21kthreadcore20:50:192
2540899201,10cyclictest6880-21idleruntime21:05:182
2540099209,8cyclictest24514-21ntpq23:40:241
2540099207,10cyclictest14905-21cron23:55:011
2540099205,7cyclictest0-21swapper/119:30:121
2540099202,12cyclictest22703-21ssh00:35:131
2540099201,9cyclictest0-21swapper/122:15:171
25400992014,4cyclictest0-21swapper/120:20:171
25400992010,3cyclictest11203-21diskmemload21:30:311
2540099200,17cyclictest9503-21latency_hist00:10:031
2540099200,14cyclictest2881-21ls19:50:281
2541699198,8cyclictest0-21swapper/322:24:473
2541699193,11cyclictest0-21swapper/323:10:263
2541699191,13cyclictest0-21swapper/319:45:013
2541699191,12cyclictest0-21swapper/320:10:013
25416991910,6cyclictest35966irq/58-eth021:10:143
25416991910,6cyclictest0-21swapper/300:12:183
2540899191,9cyclictest25443-21latency_hist19:10:012
2540899191,8cyclictest9900-21kthreadcore20:00:192
2540899191,8cyclictest9900-21kthreadcore20:00:192
2540899191,8cyclictest2017-21fschecks_time21:00:162
2540899191,10cyclictest7833-21ls19:25:162
2540899191,10cyclictest23342-21diskstats20:15:152
2540099193,3cyclictest131rcu_preempt20:20:011
2540099193,3cyclictest131rcu_preempt19:45:151
2540099191,9cyclictest0-21swapper/123:55:531
2540099191,9cyclictest0-21swapper/123:55:531
2540099191,5cyclictest27845-21apt-get22:50:021
25400991911,6cyclictest26352-21sshd22:48:131
2540099191,15cyclictest5837-21apt-get21:25:131
2540099191,15cyclictest29902-21diskstats23:45:171
2540099191,10cyclictest29446-21/usr/sbin/munin20:55:191
2540099191,10cyclictest16374-21irqstats22:25:201
25400991910,3cyclictest11203-21diskmemload22:56:501
2541699189,6cyclictest0-21swapper/322:05:223
2541699183,9cyclictest0-21swapper/320:32:513
2541699182,11cyclictest0-21swapper/319:50:153
2541699181,11cyclictest0-21swapper/320:55:313
25416991810,6cyclictest0-21swapper/300:30:053
2540099189,6cyclictest7951-21kthreadcore22:20:211
2540099188,7cyclictest6546-21/usr/sbin/munin23:30:261
2540099185,2cyclictest131rcu_preempt00:11:341
2540099181,9cyclictest6914-21cut19:55:211
2540099181,9cyclictest24260-21ssh00:00:121
2540099180,14cyclictest23677-21ssh21:35:121
2541699179,5cyclictest35966irq/58-eth021:45:143
2541699178,6cyclictest35966irq/58-eth000:20:143
2541699173,7cyclictest0-21swapper/321:20:123
2541699172,9cyclictest0-21swapper/321:05:153
2541699172,10cyclictest0-21swapper/322:50:043
2541699172,10cyclictest0-21swapper/320:25:113
2541699172,10cyclictest0-21swapper/319:10:133
2541699172,10cyclictest0-21swapper/300:35:133
2541699171,11cyclictest0-21swapper/320:40:013
25416991710,3cyclictest0-21swapper/321:38:543
2540099179,6cyclictest25487-21ntpq19:40:251
2540099179,6cyclictest19264-21tr21:50:191
2540099179,5cyclictest3523-21cut19:20:181
2540099178,7cyclictest1414-21ntp_states19:15:241
2540099177,7cyclictest4968-21cron20:30:011
2540099175,8cyclictest21-21ksoftirqd/121:05:141
2540099171,13cyclictest6424-21munin-run21:05:011
2540099171,13cyclictest27930-21wc00:20:141
2541699168,6cyclictest0-21swapper/322:55:513
2541699162,9cyclictest0-21swapper/320:45:053
2541699162,10cyclictest0-21swapper/300:15:153
2540099169,4cyclictest12697-21unixbench_multi20:00:291
2540099169,4cyclictest12697-21unixbench_multi20:00:291
2540099168,6cyclictest20111-21if_eth020:45:181
2540099163,4cyclictest21-21ksoftirqd/120:10:281
2540099162,11cyclictest14022-21irqcore20:05:171
2540099161,12cyclictest3314-21cut20:25:191
2540099161,11cyclictest18838-21sed20:40:301
2540099161,10cyclictest7725-21fschecks_count19:25:151
2541699152,9cyclictest0-21swapper/321:00:143
2541699152,9cyclictest0-21swapper/320:50:123
2541699151,12cyclictest0-21swapper/319:45:153
2541699149,3cyclictest0-21swapper/319:15:173
2541699141,9cyclictest0-21swapper/319:20:133
2541699141,8cyclictest0-21swapper/320:40:193
2541699141,11cyclictest0-21swapper/319:35:013
2541699141,11cyclictest0-21swapper/319:30:023
2540099141,10cyclictest25434-21sort19:10:011
2541699122,6cyclictest0-21swapper/319:55:133
2541699121,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional