You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 11:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 28, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2504399643,44cyclictest26810-21ssh22:30:140
25051995950,3cyclictest13791-21sh22:05:121
25071995536,12cyclictest35966irq/58-eth023:10:133
2504399553,43cyclictest27865-21ssh00:00:140
2505999522,40cyclictest0-21swapper/220:00:132
25043995231,6cyclictest12-21ksoftirqd/023:55:110
2504399494,33cyclictest31817-21ssh00:20:140
2507199482,27cyclictest35966irq/58-eth022:00:133
2505999483,4cyclictest27346irq/40-dwc2_hso21:40:132
2505999474,34cyclictest27880-21ssh00:00:152
2505199412,32cyclictest30572-21ssh22:50:141
2507199401,35cyclictest0-21swapper/321:15:143
2504399392,5cyclictest131rcu_preempt21:20:130
2504399381,27cyclictest22843-21ssh22:10:110
2507199373,26cyclictest35966irq/58-eth000:05:143
2507199371,32cyclictest0-21swapper/300:30:133
2505999372,19cyclictest17870-21ssh23:17:502
2505999372,18cyclictest14106-21sh22:57:522
2505999371,29cyclictest0-21swapper/222:30:132
25071993623,8cyclictest35966irq/58-eth021:20:123
2507199362,20cyclictest35966irq/58-eth000:10:133
25059993521,8cyclictest261rcuc/222:25:112
2504399352,28cyclictest12489-21ssh22:40:130
2504399351,29cyclictest6004-21ssh21:25:130
2505199341,23cyclictest28114-21rm21:55:131
2505999332,15cyclictest0-21swapper/200:25:452
2505199331,25cyclictest16304-21ssh23:00:121
2507199321,28cyclictest35966irq/58-eth000:15:143
2507199321,27cyclictest0-21swapper/321:30:133
2505999321,15cyclictest841-21ssh00:37:442
2507199316,21cyclictest0-21swapper/321:10:143
2507199316,21cyclictest0-21swapper/321:10:133
2507199313,23cyclictest0-21swapper/321:55:133
25071993124,4cyclictest35966irq/58-eth021:25:123
2507199310,28cyclictest0-21swapper/322:10:113
2505199311,8cyclictest22361-21latency_hist00:15:011
2507199302,22cyclictest35966irq/58-eth022:40:133
2507199302,20cyclictest35966irq/58-eth023:15:133
25071993018,8cyclictest35966irq/58-eth023:45:153
25071993018,8cyclictest35966irq/58-eth023:45:143
2507199301,20cyclictest0-21swapper/322:35:103
2507199300,28cyclictest0-21swapper/323:35:143
2507199300,23cyclictest0-21swapper/323:25:133
2507199300,23cyclictest0-21swapper/323:25:123
2505999304,11cyclictest0-21swapper/219:10:142
25043993021,4cyclictest12-21ksoftirqd/021:15:150
2507199292,24cyclictest35966irq/58-eth000:20:133
2507199291,25cyclictest0-21swapper/319:45:133
2505999293,10cyclictest0-21swapper/200:33:042
2505999292,11cyclictest29010-21ssh23:40:442
2505999291,22cyclictest1-21systemd00:17:592
25059992911,14cyclictest27-21ksoftirqd/221:45:142
2505199293,18cyclictest4535-21ssh00:05:131
2505199291,18cyclictest32060-21sh22:15:151
2505199291,18cyclictest32060-21sh22:15:141
2504399292,21cyclictest2660-21ssh23:26:460
2504399292,21cyclictest2660-21ssh23:26:450
25043992917,8cyclictest151rcuc/021:10:130
25043992917,8cyclictest151rcuc/021:10:130
2507199281,22cyclictest0-21swapper/320:40:133
2507199281,21cyclictest0-21swapper/321:00:103
2507199281,19cyclictest0-21swapper/323:05:133
2507199280,25cyclictest0-21swapper/321:35:123
2505999283,16cyclictest27-21ksoftirqd/222:10:112
2505999282,17cyclictest27346irq/40-dwc2_hso20:40:132
2505199284,10cyclictest21-21ksoftirqd/100:20:261
2505199283,19cyclictest5805-21rm23:30:101
2504399283,19cyclictest23909-21ssh23:40:110
2507199271,23cyclictest0-21swapper/321:05:123
2507199271,23cyclictest0-21swapper/300:25:133
2507199270,23cyclictest0-21swapper/321:50:143
2507199270,23cyclictest0-21swapper/321:50:143
2505999272,21cyclictest0-21swapper/223:55:452
2505999272,19cyclictest0-21swapper/220:55:142
25059992717,4cyclictest261rcuc/221:05:132
2505999271,11cyclictest10823-21diskmemload23:25:462
2505999271,11cyclictest10823-21diskmemload23:25:452
2505999270,21cyclictest10823-21diskmemload22:35:102
2505199278,5cyclictest21-21ksoftirqd/121:55:011
2505199278,5cyclictest21-21ksoftirqd/121:55:001
2505199272,19cyclictest32378-21ssh23:44:451
25051992720,3cyclictest2100-21ssh23:10:141
25051992716,5cyclictest26795-21apt-get22:30:131
2505199271,18cyclictest21572-21ssh22:45:161
2504399273,3cyclictest131rcu_preempt20:05:120
2504399272,19cyclictest14879-21ssh23:35:140
2507199262,22cyclictest0-21swapper/319:25:133
2507199261,21cyclictest0-21swapper/323:55:113
2507199261,20cyclictest0-21swapper/322:05:133
2507199261,20cyclictest0-21swapper/319:15:153
2505999269,6cyclictest27-21ksoftirqd/223:45:142
2505999269,6cyclictest27-21ksoftirqd/223:45:142
2505999263,16cyclictest27346irq/40-dwc2_hso22:40:132
2505999262,18cyclictest0-21swapper/219:40:142
25059992613,6cyclictest21886-21ssh21:32:492
2505199269,4cyclictest21-21ksoftirqd/122:22:341
2505199262,19cyclictest568-21ssh23:45:161
2505199262,19cyclictest568-21ssh23:45:151
25051992613,10cyclictest4282-21ssh00:05:021
2505199261,17cyclictest22723-21ssh00:15:151
2504399267,8cyclictest0-21swapper/000:35:130
2507199251,20cyclictest0-21swapper/322:55:133
2507199251,19cyclictest0-21swapper/320:50:103
2507199250,20cyclictest0-21swapper/322:30:133
2505999257,5cyclictest13298-21apt-get20:05:132
2505999252,18cyclictest13408-21ssh23:33:462
2505199257,6cyclictest21-21ksoftirqd/121:10:181
2505199257,6cyclictest21-21ksoftirqd/121:10:181
2505199255,10cyclictest21-21ksoftirqd/122:41:301
2505199252,4cyclictest21-21ksoftirqd/121:45:191
25051992517,5cyclictest21743-21apt-get19:40:121
25051992515,5cyclictest16941-21apt-get19:35:121
25051992511,4cyclictest21-21ksoftirqd/121:31:451
2507199243,16cyclictest0-21swapper/320:05:123
2507199243,15cyclictest0-21swapper/300:37:453
2507199242,19cyclictest0-21swapper/320:15:143
2507199241,20cyclictest0-21swapper/323:00:283
2507199241,20cyclictest0-21swapper/320:25:123
2507199241,19cyclictest0-21swapper/320:00:133
2507199240,21cyclictest0-21swapper/322:50:133
2507199240,21cyclictest0-21swapper/322:25:113
2507199240,21cyclictest0-21swapper/319:40:123
2507199240,20cyclictest0-21swapper/319:35:133
2507199240,19cyclictest0-21swapper/320:10:133
2505999241,14cyclictest3036-21ssh00:20:252
25059992410,11cyclictest3023-21vmstat22:50:322
2505199246,5cyclictest21-21ksoftirqd/123:38:501
2505199245,5cyclictest21-21ksoftirqd/120:15:301
2505199244,5cyclictest21-21ksoftirqd/121:25:191
2505199243,4cyclictest21-21ksoftirqd/122:35:191
2505199242,16cyclictest24132-21ssh21:35:121
25051992419,3cyclictest201rcuc/122:11:071
2505199241,4cyclictest21-21ksoftirqd/122:05:011
25051992414,5cyclictest201rcuc/122:28:491
2505199241,19cyclictest30005-21apt-get19:15:151
2505199241,17cyclictest25518-21ssh23:05:141
2504399243,13cyclictest20113-21ssh23:20:120
25043992415,7cyclictest8684-21ssh22:20:150
2504399241,15cyclictest8149-21ssh00:05:260
2507199231,19cyclictest0-21swapper/323:50:233
2507199231,19cyclictest0-21swapper/323:30:103
2507199231,19cyclictest0-21swapper/322:20:123
2507199231,19cyclictest0-21swapper/320:20:113
2507199231,18cyclictest0-21swapper/323:40:113
2507199231,17cyclictest0-21swapper/322:15:123
2507199231,17cyclictest0-21swapper/322:15:123
2507199230,21cyclictest0-21swapper/320:55:123
2507199230,20cyclictest0-21swapper/323:20:153
2507199230,20cyclictest0-21swapper/319:50:123
2507199230,19cyclictest0-21swapper/319:55:123
2505999232,6cyclictest0-21swapper/223:37:562
2505999232,16cyclictest0-21swapper/223:52:542
2505999232,16cyclictest0-21swapper/223:24:452
2505999232,16cyclictest0-21swapper/223:00:452
2505999232,16cyclictest0-21swapper/200:12:582
2505999231,14cyclictest2914-21kthreadcore21:00:172
2505199238,8cyclictest21-21ksoftirqd/100:27:251
2505199236,9cyclictest21-21ksoftirqd/120:25:161
2505199236,8cyclictest21-21ksoftirqd/123:19:351
2505199234,4cyclictest21-21ksoftirqd/100:35:301
2505199233,5cyclictest21-21ksoftirqd/123:28:011
2505199233,5cyclictest21-21ksoftirqd/123:28:011
2505199232,4cyclictest21-21ksoftirqd/120:55:221
25051992318,3cyclictest201rcuc/123:50:261
25051992318,3cyclictest201rcuc/121:43:241
25051992318,3cyclictest201rcuc/121:22:351
25051992318,3cyclictest201rcuc/121:16:281
25051992311,10cyclictest0-21swapper/123:23:021
2504399236,10cyclictest17486-21ssh00:30:130
25043992317,3cyclictest0-21swapper/023:50:140
25043992316,4cyclictest9301-21grep23:30:200
2507199221,19cyclictest0-21swapper/300:00:133
2507199221,18cyclictest0-21swapper/322:47:443
2507199221,18cyclictest0-21swapper/321:40:203
2507199221,18cyclictest0-21swapper/320:30:193
2507199220,20cyclictest0-21swapper/321:45:143
2507199220,20cyclictest0-21swapper/320:45:143
2507199220,19cyclictest0-21swapper/319:30:233
2507199220,19cyclictest0-21swapper/319:20:113
2505999222,14cyclictest27346irq/40-dwc2_hso19:15:142
2505999221,6cyclictest25000-21apt-get23:05:022
25059992211,9cyclictest30136-21ssh22:50:002
2505999221,13cyclictest0-21swapper/220:30:132
2505999221,10cyclictest6007-21sh21:25:122
25059992210,9cyclictest8257-21cron22:20:012
25059992210,9cyclictest8257-21cron22:20:002
25059992210,6cyclictest16754-21ssh21:11:202
25059992210,6cyclictest16754-21ssh21:11:202
2505199227,3cyclictest131rcu_preempt23:56:241
2505199226,10cyclictest28605-21ntp_192.168.11419:10:231
2505199224,5cyclictest21-21ksoftirqd/120:45:291
2505199224,4cyclictest21-21ksoftirqd/119:55:291
25051992215,5cyclictest201rcuc/100:34:011
2504399225,8cyclictest0-21swapper/019:45:140
2504399221,8cyclictest53646usb-storage22:15:050
2504399221,8cyclictest53646usb-storage22:15:040
25043992214,5cyclictest10823-21diskmemload21:50:150
25043992214,5cyclictest10823-21diskmemload21:50:140
2504399221,16cyclictest3402-21sh22:35:110
2507199210,18cyclictest0-21swapper/319:10:153
2505999219,5cyclictest24789-21cat21:15:322
2505999218,11cyclictest10823-21diskmemload00:05:462
2505999217,11cyclictest23057-21df_inode20:15:172
2505999214,7cyclictest29350-21ssh21:20:132
2505999212,14cyclictest0-21swapper/220:20:112
2505999211,11cyclictest3499-21grep19:55:012
2505999211,11cyclictest22641-21awk21:50:262
2505999211,11cyclictest22641-21awk21:50:262
25059992110,5cyclictest14747-21ssh22:21:592
2505199216,7cyclictest21-21ksoftirqd/122:58:551
2505199214,5cyclictest21-21ksoftirqd/121:05:251
2505199213,5cyclictest21-21ksoftirqd/120:50:201
2505199213,4cyclictest21-21ksoftirqd/119:20:151
2505199212,15cyclictest21-21ksoftirqd/120:35:121
25051992114,4cyclictest1424-21apt-get21:00:101
2504399219,3cyclictest131rcu_preempt22:50:210
2504399213,12cyclictest0-21swapper/019:35:050
2504399212,12cyclictest151rcuc/022:25:110
25043992112,6cyclictest14062-21hwlatdetect22:05:170
2507199201,17cyclictest0-21swapper/320:35:123
2505999208,9cyclictest4892-21cut23:10:222
2505999201,9cyclictest3710-21cpu19:55:142
2505999201,14cyclictest27346irq/40-dwc2_hso20:25:142
2505999201,10cyclictest5073-21cut19:20:212
2505999201,10cyclictest28214-21cat20:50:272
2505999201,10cyclictest25291-21tr21:35:212
2505999201,10cyclictest15812-21ssh22:05:202
2505999201,10cyclictest10905-21kthreadcore20:35:202
2505999200,6cyclictest21783-21switchtime20:10:282
2505199204,4cyclictest21-21ksoftirqd/120:40:291
2505199204,4cyclictest21-21ksoftirqd/120:10:191
2505199204,10cyclictest0-21swapper/119:25:121
2505199202,15cyclictest28716-21kthreadcore19:45:211
25051992014,4cyclictest201rcuc/119:50:161
25051992014,4cyclictest201rcuc/119:30:181
2505199200,3cyclictest91-21pr/ttyS020:10:011
2505199200,17cyclictest28030-21ls20:20:161
2504399205,11cyclictest27546-21cron20:20:010
2504399202,15cyclictest24143-21ssh21:35:120
2504399202,13cyclictest8169-21ssh23:48:460
2504399202,13cyclictest8169-21ssh23:48:450
2504399202,12cyclictest844-21munin-node21:45:150
2504399201,8cyclictest28234-21df21:55:160
25043992014,4cyclictest0-21swapper/021:30:120
25043992013,4cyclictest12263-21mailstats00:25:280
2504399200,16cyclictest22803-21ls00:15:160
2505999195,7cyclictest0-21swapper/221:55:212
2505999191,7cyclictest12458-21irqcore19:30:182
2505999191,15cyclictest5938-21kthreadcore22:00:202
2505999191,15cyclictest18488-21kthreadcore19:35:192
2505999191,10cyclictest9098-21kthreadcore19:25:202
2505999191,10cyclictest22819-21awk20:45:242
2505199193,5cyclictest21-21ksoftirqd/120:30:191
2504399196,10cyclictest15305-21cat23:15:320
2504399192,4cyclictest53646usb-storage20:10:100
2504399192,11cyclictest13509-21ssh00:10:050
2504399191,12cyclictest0-21swapper/020:55:110
25043991910,6cyclictest0-21swapper/023:05:270
2504399190,13cyclictest4857-21cpuspeed22:00:150
2505999181,9cyclictest29859-21missed_timers19:45:232
2505199182,4cyclictest201rcuc/120:00:201
2504399188,2cyclictest131rcu_preempt21:05:210
2504399187,8cyclictest25419-21hwlatdetect19:10:160
2504399182,12cyclictest19493-21tr23:00:220
25043991814,2cyclictest0-21swapper/019:30:120
2504399181,14cyclictest0-21swapper/020:00:140
2504399180,13cyclictest10648-21latency22:55:230
2505999171,9cyclictest25158-21latency_hist19:10:022
2504399177,4cyclictest11773-21munin-run19:30:000
2504399176,9cyclictest2410-21wc19:50:260
2504399172,6cyclictest9549-21munin-run20:35:010
2504399172,12cyclictest31267-21ntp_states20:20:240
2504399172,12cyclictest30948-21kthreadcore19:15:210
2504399172,12cyclictest22884-21ntp_states20:45:270
2504399172,12cyclictest11494-21kthreadcore20:35:200
25043991712,3cyclictest384-21ls20:25:170
2504399171,13cyclictest1559-21kthreadcore21:40:210
2504399171,12cyclictest2256-21fschecks_count23:10:160
2504399170,9cyclictest6367-21kthreadcore19:55:230
2507199161,13cyclictest0-21swapper/319:10:023
2504399162,9cyclictest0-21swapper/021:00:110
2504399162,11cyclictest21582-21ssh22:45:160
2504399161,12cyclictest27918-21cat20:50:260
2504399161,10cyclictest25433-21timerandwakeup19:40:270
2504399160,6cyclictest2543-21cut19:20:150
2504399151,11cyclictest14787-21cut20:40:170
2505199148,4cyclictest21-21ksoftirqd/119:10:031
2504399133,8cyclictest25047-21sed19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional