You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 12:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Apr 16, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3188999573,46cyclictest8802-21ssh22:35:121
3188999502,37cyclictest10461-21ssh23:30:121
31881994910,8cyclictest131rcu_preempt00:30:160
3188999472,37cyclictest22993-21ssh23:55:121
31881994727,8cyclictest131rcu_preempt19:10:120
3188999462,34cyclictest15897-21ssh21:45:131
31909994328,12cyclictest35966irq/58-eth022:05:123
3188199396,27cyclictest15821-21ssh23:15:120
3189799382,19cyclictest1816-21hwlatdetect21:55:162
3188199385,29cyclictest28415-21ssh23:40:120
3188199383,27cyclictest24907-21ssh21:50:110
3190999374,21cyclictest35966irq/58-eth019:10:133
3190999372,21cyclictest35966irq/58-eth020:35:133
3190999372,19cyclictest35966irq/58-eth023:15:133
31881993722,8cyclictest131rcu_preempt00:35:120
3190999367,18cyclictest35966irq/58-eth021:55:123
3190999363,23cyclictest35966irq/58-eth000:10:133
3190999361,21cyclictest35966irq/58-eth000:30:153
31889993610,18cyclictest30281-21ssh23:05:121
3190999346,17cyclictest35966irq/58-eth022:30:123
3190999346,17cyclictest35966irq/58-eth022:30:123
3190999341,20cyclictest35966irq/58-eth019:40:113
3189799342,24cyclictest24912-21ssh23:20:132
3189799341,11cyclictest880766ssh22:35:132
31881993426,6cyclictest14571-21ssh22:36:330
31881993310,17cyclictest12-21ksoftirqd/022:30:140
31881993310,17cyclictest12-21ksoftirqd/022:30:140
3190999324,18cyclictest35966irq/58-eth023:20:143
3190999322,20cyclictest35966irq/58-eth020:10:123
3190999322,16cyclictest35966irq/58-eth023:40:113
3189799323,17cyclictest27346irq/40-dwc2_hso19:55:132
31897993213,4cyclictest27-21ksoftirqd/200:30:152
3188199329,13cyclictest12-21ksoftirqd/022:00:130
31881993216,11cyclictest12382-21ssh21:25:130
31881993210,11cyclictest12-21ksoftirqd/021:35:140
31881993210,11cyclictest12-21ksoftirqd/021:35:130
31897993123,5cyclictest30335-21ssh00:35:132
31897993121,2cyclictest27346irq/40-dwc2_hso21:20:122
3188999313,18cyclictest844-21munin-node19:15:121
31889993122,3cyclictest0-21swapper/121:20:121
3188199316,22cyclictest12-21ksoftirqd/022:45:130
3188199315,18cyclictest12-21ksoftirqd/022:50:110
3188199314,20cyclictest23009-21ssh23:55:130
3190999303,9cyclictest0-21swapper/320:20:123
3190999303,19cyclictest35966irq/58-eth021:00:143
3190999303,17cyclictest35966irq/58-eth022:20:113
3189799302,22cyclictest3307-21ssh00:20:132
3189799302,21cyclictest0-21swapper/220:35:132
3188199301,23cyclictest0-21swapper/020:15:130
3188199300,23cyclictest0-21swapper/023:45:110
3190999292,17cyclictest35966irq/58-eth000:00:123
3190999292,17cyclictest35966irq/58-eth000:00:113
31909992914,8cyclictest0-21swapper/320:05:133
3189799293,18cyclictest27346irq/40-dwc2_hso19:20:122
3189799292,22cyclictest22485-21ssh22:24:342
3188999291,22cyclictest9793-21ssh21:59:331
3188199292,23cyclictest17858-21ssh22:40:110
3190999282,6cyclictest0-21swapper/320:00:113
3190999281,5cyclictest36066irq/59-eth021:20:123
3190999281,5cyclictest0-21swapper/321:50:113
3190999281,3cyclictest0-21swapper/300:05:143
3189799282,20cyclictest7060-21ssh00:03:342
3189799282,20cyclictest7060-21ssh00:03:342
3189799282,20cyclictest0-21swapper/219:30:132
3188999281,17cyclictest21319-21ssh23:00:151
3188999281,17cyclictest21319-21ssh23:00:141
3190999272,4cyclictest0-21swapper/322:55:023
3190999271,5cyclictest0-21swapper/322:55:513
3190999271,5cyclictest0-21swapper/321:45:123
3189799274,13cyclictest27346irq/40-dwc2_hso23:30:122
3189799273,5cyclictest0-21swapper/223:00:142
3189799273,5cyclictest0-21swapper/223:00:142
3189799272,18cyclictest27346irq/40-dwc2_hso23:35:112
3189799271,20cyclictest0-21swapper/219:35:142
3190999263,17cyclictest0-21swapper/300:38:343
3190999262,19cyclictest0-21swapper/319:35:123
3190999261,6cyclictest0-21swapper/323:25:233
3190999261,6cyclictest0-21swapper/321:15:213
3190999261,5cyclictest0-21swapper/323:48:103
3190999261,4cyclictest0-21swapper/323:55:023
3190999261,4cyclictest0-21swapper/320:30:133
3190999261,17cyclictest0-21swapper/321:25:133
3190999261,12cyclictest35966irq/58-eth021:10:133
3189799263,13cyclictest23242-21ssh22:25:112
3189799262,9cyclictest0-21swapper/220:20:142
31889992616,5cyclictest21-21ksoftirqd/119:20:121
3188199263,20cyclictest12293-21ssh00:25:130
3188199262,19cyclictest28702-21ssh22:10:130
3188199262,18cyclictest3353-21ssh21:20:150
3188199262,17cyclictest1558-21ssh21:55:130
3188199261,20cyclictest1562-21ssh23:25:120
3190999257,3cyclictest0-21swapper/323:04:053
3190999257,3cyclictest0-21swapper/323:04:053
3190999257,14cyclictest35966irq/58-eth000:20:133
3190999253,2cyclictest0-21swapper/322:35:263
3190999252,16cyclictest35966irq/58-eth022:00:123
3190999251,5cyclictest0-21swapper/322:10:223
3190999251,5cyclictest0-21swapper/319:30:173
3190999251,5cyclictest0-21swapper/300:15:243
3190999251,4cyclictest0-21swapper/300:00:013
3190999251,3cyclictest0-21swapper/323:10:203
3190999251,15cyclictest35966irq/58-eth023:30:123
3190999250,15cyclictest35966irq/58-eth020:15:133
3189799252,6cyclictest0-21swapper/221:53:342
31897992513,10cyclictest13147-21cut22:55:192
31897992512,9cyclictest0-21swapper/220:55:192
31897992512,4cyclictest0-21swapper/200:10:122
3189799251,20cyclictest21359-21cut20:40:142
3188199253,16cyclictest19586-21ssh22:05:120
3188199252,20cyclictest6900-21ssh23:10:130
3190999248,10cyclictest35966irq/58-eth023:05:143
3190999242,13cyclictest0-21swapper/322:40:133
3190999241,9cyclictest0-21swapper/321:40:213
3190999241,5cyclictest0-21swapper/323:37:333
3190999241,5cyclictest0-21swapper/322:48:293
3190999241,5cyclictest0-21swapper/322:26:203
3190999241,5cyclictest0-21swapper/321:35:103
3190999241,5cyclictest0-21swapper/321:35:093
3190999241,5cyclictest0-21swapper/321:30:223
3190999241,5cyclictest0-21swapper/321:30:213
3190999241,5cyclictest0-21swapper/321:05:273
3190999241,5cyclictest0-21swapper/319:45:193
3190999241,5cyclictest0-21swapper/319:45:193
3190999241,5cyclictest0-21swapper/300:28:593
3190999241,4cyclictest0-21swapper/322:15:193
3190999241,4cyclictest0-21swapper/319:50:163
3189799247,13cyclictest1530-21apt-get23:25:112
3189799242,18cyclictest17844-21apt-get22:40:112
3189799242,16cyclictest0-21swapper/222:45:272
3189799241,19cyclictest0-21swapper/223:05:142
31897992410,5cyclictest20302-21kernelversion22:05:192
3188999243,14cyclictest844-21munin-node20:35:131
3188999241,17cyclictest30719-21ssh21:51:351
3188999241,12cyclictest7019-21ntp_states22:50:241
3188199243,16cyclictest5299-21ssh22:15:130
31881992419,3cyclictest31869-21cyclictest20:05:110
31881992415,6cyclictest151rcuc/000:05:120
3188199241,3cyclictest131rcu_preempt21:30:490
3188199241,3cyclictest131rcu_preempt21:30:480
31881992413,7cyclictest17403-21diskmemload00:15:120
3190999231,6cyclictest0-21swapper/320:55:163
3190999231,6cyclictest0-21swapper/319:25:193
3190999231,6cyclictest0-21swapper/319:20:123
3190999231,5cyclictest0-21swapper/320:45:283
3190999231,5cyclictest0-21swapper/320:40:163
3190999231,5cyclictest0-21swapper/320:00:023
3189799233,16cyclictest27346irq/40-dwc2_hso19:40:122
3189799232,6cyclictest0-21swapper/223:59:342
3189799232,6cyclictest0-21swapper/222:32:352
3189799232,6cyclictest0-21swapper/222:32:342
3189799231,9cyclictest19358-21ssh21:45:212
31897992314,7cyclictest19531-21rm23:15:272
31897992314,6cyclictest0-21swapper/222:50:122
3189799231,13cyclictest14015-21ssh23:50:132
31897992311,10cyclictest11402-21cron20:30:002
3188999238,12cyclictest23413-1kworker/u9:223:20:131
3188999232,17cyclictest12307-21ssh22:55:131
3188999231,8cyclictest32025-21ntpq23:40:231
31889992313,3cyclictest21-21ksoftirqd/120:25:121
3188999231,18cyclictest17874-21ssh22:40:121
3190999221,5cyclictest0-21swapper/320:50:173
3190999221,4cyclictest0-21swapper/319:15:203
3189799229,10cyclictest0-21swapper/200:25:152
3189799227,12cyclictest10628-21kthreadcore00:05:222
3189799222,6cyclictest0-21swapper/222:01:342
3189799221,14cyclictest19892-21ssh21:10:202
3189799221,13cyclictest28637-21fschecks_time23:40:152
31897992210,9cyclictest17403-21diskmemload21:10:012
31897992210,5cyclictest3457-21ssh21:36:202
31897992210,5cyclictest3457-21ssh21:36:202
3188999227,3cyclictest17403-21diskmemload22:24:521
3188999222,15cyclictest17403-21diskmemload00:20:131
3188999221,16cyclictest32425-21tr00:00:161
3188999221,16cyclictest32425-21tr00:00:151
3188999221,10cyclictest12934-21cut21:25:171
3188199223,15cyclictest26765-21ssh21:15:140
3188199221,3cyclictest131rcu_preempt19:25:280
31881992213,7cyclictest17891-21grep00:10:150
3188199221,15cyclictest3532-21tr20:55:180
3190999211,6cyclictest0-21swapper/320:25:163
3189799218,10cyclictest1511-21ssh00:18:222
3189799211,17cyclictest5616-21cut23:45:172
3189799211,15cyclictest0-21swapper/220:50:132
3189799211,11cyclictest21808-21irqcore21:30:172
3189799211,11cyclictest21808-21irqcore21:30:172
3189799211,11cyclictest19018-21tail20:00:282
3189799211,10cyclictest15665-21ls21:45:012
31897992110,9cyclictest2021-21sshd22:11:342
3188999213,13cyclictest628-21tcf-agent23:35:131
3188999211,9cyclictest28364-21sed22:10:011
3188999211,8cyclictest30532-21/usr/sbin/munin00:15:271
3188999211,14cyclictest17707-21sh00:10:121
31889992110,5cyclictest21-21ksoftirqd/120:45:111
3188199219,3cyclictest0-21swapper/020:10:160
3188199213,12cyclictest0-21swapper/021:00:140
3188199211,17cyclictest4329-21apt-get19:15:110
3188199211,16cyclictest825-21apt-get19:45:120
3188199211,16cyclictest825-21apt-get19:45:120
3188199211,14cyclictest6897-21ssh21:40:130
3190999201,5cyclictest0-21swapper/319:10:013
3189799208,9cyclictest6849-21/usr/sbin/munin23:10:112
3189799203,9cyclictest947-21fschecks_count19:45:162
3189799203,9cyclictest947-21fschecks_count19:45:152
3189799202,4cyclictest14999-21ssh21:25:212
3189799201,9cyclictest8434-21kthreadcore22:15:222
3189799201,9cyclictest25447-21cut20:10:172
3189799201,8cyclictest14663-21kthreadcore20:30:222
3189799201,16cyclictest5975-21ps19:50:172
3189799201,10cyclictest31974-21/usr/sbin/munin19:10:222
3189799201,10cyclictest31954-21latency_hist19:10:012
3189799200,16cyclictest26729-21apt-get21:15:122
3188999205,3cyclictest131rcu_preempt22:45:201
3188999204,3cyclictest17403-21diskmemload23:47:261
3188999202,6cyclictest10843-21kthreadcore00:05:201
3188999201,16cyclictest25317-21ls21:30:281
3188999201,16cyclictest25317-21ls21:30:281
3188999201,15cyclictest31511-21kthreadcore20:50:181
3188199202,15cyclictest21250-21apt-get20:40:110
31881992013,5cyclictest13480-21kthreadcore23:30:220
31881992012,5cyclictest17791-21ssh21:10:130
3188199201,16cyclictest30273-21apt-get23:05:140
3189799198,3cyclictest27540-21kthreadcore20:45:192
3189799192,8cyclictest23429-21missed_timers20:05:222
3189799191,9cyclictest32532-21kthreadcore20:15:212
3189799191,2cyclictest0-21swapper/219:15:122
3189799191,10cyclictest16507-21kthreadcore19:25:222
3188999194,3cyclictest17403-21diskmemload00:30:221
3188999192,12cyclictest0-21swapper/119:55:131
3188999192,12cyclictest0-21swapper/119:30:141
3188999192,11cyclictest6359-21ssh22:32:351
3188999192,11cyclictest6359-21ssh22:32:341
3188999191,4cyclictest131rcu_preempt22:20:021
3188999190,16cyclictest32308-21irqcore19:10:181
3188199194,8cyclictest14261-21ssh22:20:130
3188199193,10cyclictest15908-21ssh21:45:120
3188199191,8cyclictest23425-21fschecks_count22:25:140
3188199191,5cyclictest0-21swapper/019:50:130
31881991912,5cyclictest12-21ksoftirqd/019:30:300
31881991911,6cyclictest0-21swapper/019:25:020
3188199191,15cyclictest26035-21apt-get20:45:110
3188199191,15cyclictest22975-21ssh23:35:220
3188199191,14cyclictest16444-21sed22:55:290
3189799181,9cyclictest8264-21grep21:00:162
3188999189,6cyclictest4513-21threads19:45:281
3188999189,6cyclictest4513-21threads19:45:271
3188999184,3cyclictest131rcu_preempt21:11:111
3188999183,3cyclictest131rcu_preempt19:30:001
3188999182,11cyclictest0-21swapper/122:25:131
3188999182,10cyclictest0-21swapper/120:20:131
3188999181,9cyclictest0-21swapper/121:18:371
3188999181,4cyclictest0-21swapper/123:53:311
31889991812,4cyclictest29903-21grep20:15:161
3188999181,10cyclictest0-21swapper/122:00:341
31889991810,6cyclictest24814-21cat20:40:271
31889991810,6cyclictest0-21swapper/122:13:571
3188199188,7cyclictest12-21ksoftirqd/023:20:250
3188199187,8cyclictest16700-21grep21:05:290
3188199185,10cyclictest32262-21ssh00:00:150
3188199185,10cyclictest32262-21ssh00:00:140
3188199183,13cyclictest18815-21kthreadcore20:35:220
3188199181,7cyclictest7063-21grep20:25:160
3188199181,3cyclictest131rcu_preempt23:50:320
3188199181,15cyclictest17072-21kthreadcore20:00:190
31881991810,6cyclictest12259-21ps19:55:190
3188999179,6cyclictest10632-21open_inodes21:40:251
3188999179,3cyclictest0-21swapper/121:05:141
3188999178,6cyclictest13786-21cat00:25:191
3188999175,3cyclictest17403-21diskmemload21:36:191
3188999175,3cyclictest17403-21diskmemload21:36:191
3188999172,10cyclictest26736-21kworker/u8:023:25:111
31889991711,4cyclictest0-21swapper/120:35:001
31889991711,3cyclictest6557-21latency_hist00:40:021
3188999171,12cyclictest24432-21munin-run23:20:011
3188199177,7cyclictest23469-21kthreadcore23:00:220
3188199177,7cyclictest23469-21kthreadcore23:00:210
3188199176,2cyclictest12-21ksoftirqd/020:55:010
3188199171,5cyclictest31858-21ntp_192.168.11419:40:230
3188199171,3cyclictest131rcu_preempt20:30:160
3188199171,12cyclictest3416-21diskstats00:20:150
31881991710,4cyclictest26170-21kthreadcore19:35:200
3188199170,13cyclictest2181-21uniq20:20:150
3188999162,9cyclictest28794-21tr19:40:171
3188999162,10cyclictest5739-21diskstats19:50:131
3188999161,11cyclictest28758-21unixbench_singl20:10:281
3188999161,10cyclictest0-21swapper/121:00:141
31889991610,4cyclictest0-21swapper/119:35:151
3188999160,12cyclictest73-21kswapd023:10:141
3188999152,9cyclictest0-21swapper/120:55:141
3188999151,11cyclictest20429-21cut20:05:171
3188999151,11cyclictest15680-21sed20:00:171
3188199141,10cyclictest31899-21cat19:10:010
3188999120,10cyclictest31869-21cyclictest19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional