You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 04:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 29, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
911199836,12cyclictest12-21ksoftirqd/019:40:130
912899648,11cyclictest27-21ksoftirqd/222:40:142
9128996316,23cyclictest27-21ksoftirqd/200:05:132
912899557,13cyclictest131rcu_preempt00:25:132
9111995212,7cyclictest131rcu_preempt20:55:130
911199494,34cyclictest24030-21ssh22:20:120
913899463,8cyclictest35966irq/58-eth023:25:143
912899466,23cyclictest27-21ksoftirqd/200:00:142
911999442,39cyclictest3966-21ssh22:45:131
911999441,35cyclictest31397-21ssh21:30:141
911999432,36cyclictest22360-21ssh21:25:141
911199435,5cyclictest131rcu_preempt20:30:140
911999421,33cyclictest7923-21ssh21:35:131
911999421,33cyclictest7923-21ssh21:35:131
911199426,10cyclictest12-21ksoftirqd/021:40:130
911999413,34cyclictest30329-21ssh00:30:141
9119994126,10cyclictest18486-21apt-get22:35:141
911999402,32cyclictest8924-21ssh00:00:141
913899394,23cyclictest0-21swapper/322:45:133
9138993927,5cyclictest35966irq/58-eth023:45:143
9119993910,10cyclictest29570-21apt-get22:05:141
9119993910,10cyclictest29570-21apt-get22:05:141
911199384,27cyclictest29599-21ssh22:05:150
911199384,27cyclictest29599-21ssh22:05:150
913899376,15cyclictest35966irq/58-eth023:10:143
913899376,15cyclictest35966irq/58-eth023:10:143
9138993732,3cyclictest35966irq/58-eth022:10:133
911999377,11cyclictest5470-21/usr/sbin/munin23:20:201
9119993731,3cyclictest5937-21apt-get19:40:141
911199377,4cyclictest12-21ksoftirqd/023:15:130
913899364,22cyclictest35966irq/58-eth021:00:133
913899361,9cyclictest0-21swapper/323:50:133
913899361,30cyclictest35966irq/58-eth021:10:143
9128993623,4cyclictest261rcuc/221:10:142
912899362,28cyclictest9660-21rm23:23:542
912899362,23cyclictest25894-21ssh21:45:132
911999369,10cyclictest21168-21ssh21:24:041
9119993628,4cyclictest6137-21apt-get22:10:121
9119993618,13cyclictest27846-21ssh21:10:151
912899352,27cyclictest21757-21sh23:11:052
912899352,27cyclictest21757-21sh23:11:052
912899351,16cyclictest0-21swapper/220:20:132
911999357,11cyclictest17746-21awk19:15:271
9119993530,3cyclictest12010-21apt-get20:20:131
9119993528,3cyclictest7195-21apt-get20:15:131
9119993526,4cyclictest21382-21apt-get00:25:131
9119993521,10cyclictest0-21swapper/122:55:131
9119993512,9cyclictest28506-21needreboot20:00:231
9119993510,15cyclictest2687-21ssh22:43:561
912899343,24cyclictest32369-21sh23:55:142
911999349,7cyclictest21463-21ntp_192.168.11421:00:241
9119993427,3cyclictest17833-21apt-get00:05:141
9119993423,7cyclictest7440-21apt-get23:05:121
9119993420,12cyclictest0-21swapper/121:50:131
911199341,27cyclictest2033-21ssh23:20:110
911199341,26cyclictest18513-21ssh22:35:140
913899331,19cyclictest35966irq/58-eth023:30:133
911999339,9cyclictest9935-21ssh21:16:241
911999338,10cyclictest15071-21ntp_192.168.11421:55:251
911999337,8cyclictest19566-21grep20:25:221
911999337,12cyclictest31608-21ssh23:17:051
911999336,8cyclictest350-21kthreadcore20:40:211
911999336,8cyclictest350-21kthreadcore20:40:211
911999336,11cyclictest9037-21ssh22:29:531
911999336,11cyclictest20000-21rm23:30:131
9119993327,3cyclictest1101-21apt-get19:35:141
9119993314,13cyclictest30933-21apt-get23:00:141
911199335,21cyclictest12-21ksoftirqd/021:45:120
913899323,15cyclictest35966irq/58-eth019:25:143
911999329,6cyclictest21127-21ssh22:17:021
911999327,16cyclictest19205-21/usr/sbin/munin19:20:161
911999327,11cyclictest1-21systemd00:21:541
911999327,10cyclictest13586-21grep23:25:221
911999326,11cyclictest14921-21kthreadcore22:50:241
911999326,10cyclictest990-21ntp_states20:05:251
911999326,10cyclictest12460-21ssh00:36:281
911999325,10cyclictest32461-21/usr/sbin/munin23:55:151
9119993210,8cyclictest460-21ssh23:35:291
911199325,4cyclictest131rcu_preempt21:25:140
911199323,8cyclictest12-21ksoftirqd/023:00:110
913899314,24cyclictest0-21swapper/320:20:133
911999319,17cyclictest0-21swapper/123:45:121
911999316,7cyclictest0-21swapper/123:42:371
911999316,10cyclictest30012-21ssh21:45:281
9119993125,2cyclictest26746-21apt-get00:10:141
9119993123,5cyclictest28673-21apt-get19:30:121
9119993123,3cyclictest9573-21ssh22:30:151
9119993113,14cyclictest0-21swapper/123:50:111
911999311,12cyclictest22905-21tr19:55:211
9119993110,8cyclictest29689-21/usr/sbin/munin20:35:221
9111993110,2cyclictest131rcu_preempt19:25:130
913899304,17cyclictest35966irq/58-eth021:50:153
913899303,7cyclictest0-21swapper/323:05:123
9138993015,11cyclictest35966irq/58-eth000:20:253
912899304,2cyclictest131rcu_preempt21:35:122
912899304,2cyclictest131rcu_preempt21:35:122
911999309,8cyclictest3881-21sed00:15:191
911999307,7cyclictest27442-21diskmemload23:13:251
911999307,7cyclictest27442-21diskmemload23:13:241
9119993021,3cyclictest24021-21apt-get22:20:131
9119993017,9cyclictest0-21swapper/121:40:121
912899298,12cyclictest25481-21kthreadcore22:20:202
912899293,16cyclictest930-21snmpd23:00:142
912899292,20cyclictest7467-21ssh23:05:142
9128992916,5cyclictest27-21ksoftirqd/223:15:212
911999297,8cyclictest2688-21cat20:10:171
911999296,7cyclictest0-21swapper/122:04:141
911999296,10cyclictest19140-21switchtime19:50:281
9119992922,3cyclictest13160-21apt-get20:55:151
9119992921,4cyclictest21596-21apt-get20:30:121
9119992918,8cyclictest9328-21apt-get19:10:141
912899282,19cyclictest0-21swapper/220:30:142
912899281,16cyclictest23457-21ssh23:50:132
9128992810,10cyclictest0-21swapper/222:47:272
911999286,10cyclictest11696-21missed_timers20:50:231
911199285,4cyclictest131rcu_preempt21:15:130
913899275,12cyclictest35966irq/58-eth022:05:133
913899275,12cyclictest35966irq/58-eth022:05:133
913899273,17cyclictest0-21swapper/300:30:153
9138992710,4cyclictest0-21swapper/321:29:463
9128992711,9cyclictest261rcuc/222:10:132
911999276,9cyclictest4111-21sed20:45:201
911199272,9cyclictest12-21ksoftirqd/000:00:130
913899268,12cyclictest0-21swapper/323:35:143
913899267,5cyclictest35966irq/58-eth021:55:143
913899263,16cyclictest36066irq/59-eth022:55:133
913899262,7cyclictest0-21swapper/300:05:143
913899262,15cyclictest35966irq/58-eth021:30:143
913899261,8cyclictest0-21swapper/322:30:303
913899261,11cyclictest0-21swapper/322:00:173
9138992611,13cyclictest35966irq/58-eth023:20:113
912899265,9cyclictest0-21swapper/221:40:192
912899262,19cyclictest0-21swapper/220:15:122
9128992615,4cyclictest27-21ksoftirqd/222:33:412
9128992611,5cyclictest3693-21fschecks_count20:45:172
911999265,7cyclictest0-21swapper/119:45:271
911199261,20cyclictest27442-21diskmemload00:10:140
913899259,4cyclictest0-21swapper/300:02:123
913899258,3cyclictest0-21swapper/323:17:233
913899252,11cyclictest0-21swapper/321:45:133
913899251,9cyclictest0-21swapper/322:25:313
913899251,8cyclictest0-21swapper/320:35:253
913899251,7cyclictest0-21swapper/321:00:013
9138992516,6cyclictest35966irq/58-eth000:10:143
913899251,21cyclictest0-21swapper/321:15:253
913899251,10cyclictest0-21swapper/300:15:263
912899258,10cyclictest32488-21kworker/2:200:16:202
912899257,13cyclictest27-21ksoftirqd/223:31:102
912899254,8cyclictest131rcu_preempt23:25:132
9128992514,4cyclictest27-21ksoftirqd/219:20:292
9128992513,6cyclictest27-21ksoftirqd/222:16:092
9128992513,5cyclictest27-21ksoftirqd/200:35:182
912899251,18cyclictest27346irq/40-dwc2_hso22:05:142
912899251,18cyclictest27346irq/40-dwc2_hso22:05:142
912899251,15cyclictest131rcu_preempt22:35:222
912899251,15cyclictest0-21swapper/220:05:262
911199258,4cyclictest131rcu_preempt23:10:140
911199258,4cyclictest131rcu_preempt23:10:140
911199253,15cyclictest844-21munin-node23:55:140
911199252,17cyclictest14577-21ssh23:45:130
9111992515,3cyclictest12-21ksoftirqd/021:05:120
911199251,19cyclictest27442-21diskmemload00:35:140
913899242,7cyclictest0-21swapper/322:35:223
913899242,6cyclictest0-21swapper/320:25:363
913899242,6cyclictest0-21swapper/319:55:143
913899242,5cyclictest0-21swapper/320:00:143
913899241,8cyclictest0-21swapper/321:35:203
913899241,8cyclictest0-21swapper/321:35:203
913899241,8cyclictest0-21swapper/321:20:183
913899241,7cyclictest0-21swapper/323:55:213
913899241,7cyclictest0-21swapper/322:20:213
913899241,7cyclictest0-21swapper/322:20:023
913899241,7cyclictest0-21swapper/319:10:173
913899241,7cyclictest0-21swapper/300:25:283
913899241,10cyclictest0-21swapper/319:15:153
912899246,3cyclictest27-21ksoftirqd/222:25:172
912899245,11cyclictest14582-21ssh23:45:142
912899243,11cyclictest131rcu_preempt00:33:532
912899242,12cyclictest131rcu_preempt19:35:232
9128992413,5cyclictest27-21ksoftirqd/222:01:582
912899241,20cyclictest6175-21ls19:40:172
912899241,20cyclictest1366166ssh23:44:232
912899241,19cyclictest26749-21ssh00:10:122
912899241,13cyclictest12096-21timerwakeupswit20:50:282
912899241,11cyclictest4402-21sh21:15:152
912899240,21cyclictest408-21ls23:35:292
9111992417,4cyclictest131rcu_preempt21:35:130
9111992417,4cyclictest131rcu_preempt21:35:130
913899239,3cyclictest0-21swapper/321:41:423
913899232,7cyclictest0-21swapper/300:35:183
913899232,6cyclictest0-21swapper/322:40:333
913899232,4cyclictest0-21swapper/321:10:023
913899231,7cyclictest0-21swapper/320:10:023
913899231,7cyclictest0-21swapper/319:25:023
913899231,6cyclictest0-21swapper/323:05:013
912899232,14cyclictest0-21swapper/221:51:012
912899231,4cyclictest0-21swapper/220:25:172
9128992312,5cyclictest27-21ksoftirqd/222:50:252
9128992311,8cyclictest0-21swapper/219:30:272
912899231,15cyclictest27442-21diskmemload21:23:312
912899231,15cyclictest0-21swapper/221:05:012
9128992310,6cyclictest3018-21cut20:10:192
911999231,7cyclictest0-21swapper/121:05:221
911199236,3cyclictest131rcu_preempt23:50:120
911199232,4cyclictest131rcu_preempt21:20:140
911199231,6cyclictest12-21ksoftirqd/019:50:140
9111992312,3cyclictest131rcu_preempt22:55:020
911199231,17cyclictest12-21ksoftirqd/021:50:270
913899222,6cyclictest0-21swapper/319:55:023
913899221,7cyclictest0-21swapper/322:50:263
913899221,7cyclictest0-21swapper/320:30:183
913899221,7cyclictest0-21swapper/319:35:233
913899221,6cyclictest0-21swapper/320:45:293
912899227,5cyclictest131rcu_preempt19:52:482
912899223,4cyclictest271ksoftirqd/222:57:382
912899221,5cyclictest0-21swapper/200:22:362
912899221,3cyclictest131rcu_preempt19:45:262
9128992212,5cyclictest27-21ksoftirqd/221:55:242
9128992212,2cyclictest27-21ksoftirqd/221:25:512
912899221,10cyclictest31734-21cat21:30:172
9119992210,9cyclictest24326-21cut19:25:191
911199222,7cyclictest12-21ksoftirqd/022:05:020
911199222,16cyclictest17844-21diskmemload00:05:140
9111992215,5cyclictest12-21ksoftirqd/020:25:130
911199221,19cyclictest32450-21ssh23:35:220
913899212,7cyclictest0-21swapper/323:40:243
913899211,6cyclictest0-21swapper/320:50:153
912899216,5cyclictest131rcu_preempt19:11:132
912899211,11cyclictest131rcu_preempt20:00:142
911199218,3cyclictest131rcu_preempt00:15:240
911199216,3cyclictest131rcu_preempt20:40:280
911199216,3cyclictest131rcu_preempt20:40:280
911199216,2cyclictest131rcu_preempt20:10:140
911199212,14cyclictest13474-21ssh23:06:540
9111992112,6cyclictest32466-21sh21:10:320
9111992111,8cyclictest0-21swapper/023:33:400
9111992111,8cyclictest0-21swapper/023:26:420
911199210,16cyclictest12001-21find20:20:120
912899201,8cyclictest0-21swapper/221:05:172
912899201,4cyclictest0-21swapper/220:55:282
912899201,4cyclictest0-21swapper/219:15:122
912899201,2cyclictest131rcu_preempt20:40:312
912899201,2cyclictest131rcu_preempt20:40:312
911199209,8cyclictest20059-21cron22:00:010
911199209,3cyclictest131rcu_preempt22:15:180
911199206,4cyclictest12-21ksoftirqd/022:41:450
911199204,5cyclictest7389-21kthreadcore22:10:200
911199203,3cyclictest131rcu_preempt20:00:270
911199202,3cyclictest131rcu_preempt19:35:150
9111992013,4cyclictest12-21ksoftirqd/019:20:190
9111992011,6cyclictest13850-21cat22:30:320
911199201,15cyclictest26378-21apt-get20:35:120
9111992010,2cyclictest131rcu_preempt22:46:540
913899191,12cyclictest0-21swapper/320:10:163
913899191,12cyclictest0-21swapper/319:30:123
912899191,9cyclictest26926-21irqrtprio20:35:182
912899191,3cyclictest0-21swapper/219:25:182
911199198,3cyclictest12-21ksoftirqd/022:55:180
911199196,4cyclictest131rcu_preempt23:40:310
911199195,7cyclictest12-21ksoftirqd/019:30:270
911199194,3cyclictest131rcu_preempt21:00:130
9111991913,3cyclictest12-21ksoftirqd/020:00:010
9111991913,3cyclictest12-21ksoftirqd/019:45:190
9111991912,5cyclictest0-21swapper/019:15:150
9111991912,4cyclictest12-21ksoftirqd/021:33:120
912899181,8cyclictest20972-21kernelversion19:55:192
912899181,14cyclictest9165-21latency_hist19:10:022
911999181,15cyclictest9110-21cyclictest19:10:021
911199189,3cyclictest0-21swapper/000:27:080
911199188,2cyclictest12-21ksoftirqd/000:30:170
911199181,4cyclictest131rcu_preempt00:20:170
9111991813,3cyclictest12-21ksoftirqd/020:15:200
9111991812,3cyclictest12-21ksoftirqd/022:27:240
9111991811,4cyclictest12-21ksoftirqd/020:50:210
9111991811,4cyclictest12-21ksoftirqd/019:10:160
913899173,9cyclictest35966irq/58-eth019:45:113
913899172,10cyclictest0-21swapper/319:40:143
913899171,10cyclictest0-21swapper/320:45:013
913899171,10cyclictest0-21swapper/320:45:003
911199178,2cyclictest12-21ksoftirqd/020:05:350
911199175,9cyclictest5507-21kthreadcore20:45:220
913899161,9cyclictest0-21swapper/320:15:273
911199159,3cyclictest12-21ksoftirqd/019:10:020
913899111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional