You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 08:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Wed Jul 15, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1249199604,42cyclictest25139-21ssh00:25:122
12503995843,8cyclictest35966irq/58-eth021:40:133
1247299584,9cyclictest131rcu_preempt22:15:140
12491995637,13cyclictest3876-21apt-get22:25:142
12491995637,13cyclictest3876-21apt-get22:25:132
1248199554,39cyclictest25149-21ssh00:25:141
1247299553,41cyclictest11195-21rm23:05:140
12491995424,24cyclictest14866-21ssh23:25:142
12503995330,15cyclictest35966irq/58-eth023:00:123
1247299533,7cyclictest131rcu_preempt22:50:150
1250399483,26cyclictest35966irq/58-eth000:05:133
1249199481,28cyclictest11190-21ssh21:35:142
1247299482,35cyclictest3907-21ssh22:25:150
1247299482,35cyclictest3907-21ssh22:25:140
1247299482,31cyclictest23788-21ssh23:30:130
1247299474,34cyclictest3808-21ssh23:55:140
1247299472,32cyclictest31098-21ssh22:40:140
1247299472,32cyclictest31098-21ssh22:40:130
12491994621,22cyclictest0-21swapper/222:00:132
12503994517,20cyclictest35966irq/58-eth022:05:153
12503994513,20cyclictest35966irq/58-eth021:35:143
12491994521,21cyclictest0-21swapper/221:25:132
1250399445,28cyclictest35966irq/58-eth023:25:133
12491994427,11cyclictest0-21swapper/221:55:132
12481994311,22cyclictest0-21swapper/100:05:131
12472994317,16cyclictest12-21ksoftirqd/021:50:130
1249199425,15cyclictest29266-21ssh23:15:132
12481994225,11cyclictest21946-21ssh22:35:131
12481994225,11cyclictest21946-21ssh22:35:121
1248199422,31cyclictest25583-21ssh21:25:141
12481994223,13cyclictest0-21swapper/120:15:121
1248199421,33cyclictest30540-21ssh00:10:131
12503994121,10cyclictest35966irq/58-eth021:20:133
12491994121,15cyclictest0-21swapper/222:35:132
12491994121,15cyclictest0-21swapper/222:35:122
12491994115,21cyclictest0-21swapper/200:00:152
1248199413,27cyclictest14599-21ssh21:55:141
12481994129,8cyclictest1678-21apt-get00:30:111
1247299414,30cyclictest23665-21ssh22:00:120
1250399403,21cyclictest35966irq/58-eth000:25:143
12503994012,14cyclictest35966irq/58-eth022:45:143
12491994021,16cyclictest1682-21ssh00:30:122
12481994028,8cyclictest666-21apt-get20:05:111
12491993933,3cyclictest16705-21apt-get22:50:152
12491993924,11cyclictest29111-21ssh21:45:152
12491993916,14cyclictest30591-21diskmemload22:20:142
1248199392,27cyclictest18171-21ssh23:45:121
1248199392,27cyclictest18171-21ssh23:45:111
1247299391,5cyclictest131rcu_preempt23:35:140
1250399382,20cyclictest35966irq/58-eth021:45:143
12491993831,3cyclictest9256-21apt-get22:10:142
12491993812,13cyclictest11395-21ssh00:35:192
12491993722,11cyclictest0-21swapper/220:50:132
12491993714,8cyclictest5124-21ssh00:13:062
1249199370,28cyclictest0-21swapper/200:05:142
12481993729,5cyclictest2179-21apt-get21:30:131
12481993727,7cyclictest15087-21apt-get20:20:151
12491993628,4cyclictest7667-21apt-get22:45:142
12491993626,7cyclictest20075-21apt-get21:40:152
12491993620,12cyclictest0-21swapper/223:45:122
12491993620,12cyclictest0-21swapper/223:45:112
12491993619,13cyclictest26205-21grep21:05:162
12491993616,16cyclictest11952-21kthreadcore23:05:192
12491993614,17cyclictest0-21swapper/219:50:142
12481993625,7cyclictest0-21swapper/122:15:131
12481993610,10cyclictest28173-21awk20:30:261
12472993616,11cyclictest12453-21cyclictest19:30:130
1250399354,22cyclictest35966irq/58-eth023:35:133
1250399351,20cyclictest35966irq/58-eth022:40:143
1250399351,20cyclictest35966irq/58-eth022:40:133
12491993520,12cyclictest11983-21sshd23:59:332
12491993517,12cyclictest0-21swapper/221:10:142
12491993517,10cyclictest10986-21latency_hist23:05:032
12491993514,17cyclictest6908-21fschecks_count20:45:162
12491993514,10cyclictest14238-21ssh00:18:092
1248199359,10cyclictest22034-21sh21:21:141
1248199353,27cyclictest7524-21ssh21:15:141
12481993527,4cyclictest12597-21apt-get19:10:141
12472993520,10cyclictest12-21ksoftirqd/023:50:130
1249199349,11cyclictest0-21swapper/223:32:272
12491993421,9cyclictest0-21swapper/219:40:112
12491993418,8cyclictest7860-21sed19:35:282
12491993418,8cyclictest11628-21grep21:15:302
12491993417,9cyclictest21981-21cut21:00:202
12491993417,13cyclictest805-21cat20:35:292
12491993417,13cyclictest19946-21mailstats20:55:302
12491993415,7cyclictest32680-21usage_EthcTask20:00:322
12491993415,16cyclictest18050-21ps19:15:192
12491993412,7cyclictest18985-21/usr/sbin/munin20:20:302
12491993412,16cyclictest2575466ssh22:18:352
12481993428,2cyclictest21193-21apt-get21:00:151
12491993319,5cyclictest13963-21diskstats19:45:162
12491993318,12cyclictest1319-21rm22:59:312
12491993317,14cyclictest2643-21irqstats20:40:192
12491993317,13cyclictest20927-21ssh00:20:312
12491993315,14cyclictest20356-21ssh23:10:172
12491993314,7cyclictest30591-21diskmemload21:52:072
12491993313,15cyclictest30612-21apt-get22:40:042
12491993313,15cyclictest30612-21apt-get22:40:032
1248199337,10cyclictest9722-21ssh23:56:471
12481993327,3cyclictest5629-21apt-get21:50:131
12481993326,4cyclictest17379-21apt-get19:15:131
12481993312,14cyclictest21-21ksoftirqd/120:35:141
12481993310,8cyclictest30591-21diskmemload23:00:091
12491993216,8cyclictest14609-21ssh23:41:092
12491993216,6cyclictest16741-21cat19:10:302
12491993216,13cyclictest19444-21ssh22:32:302
12491993216,12cyclictest2159-21/usr/sbin/munin21:30:132
12491993216,10cyclictest0-21swapper/223:35:142
1248199329,8cyclictest4470-21ssh23:18:471
1248199329,8cyclictest15489-21ssh22:12:101
1248199328,8cyclictest16262-21ssh21:35:461
1248199325,11cyclictest14091-21sh23:24:341
12481993226,3cyclictest4252-21apt-get19:35:131
12481993226,2cyclictest16188-21apt-get00:20:131
12481993224,3cyclictest32703-21apt-get23:35:131
12481993221,7cyclictest0-21swapper/100:00:131
12481993218,10cyclictest0-21swapper/123:10:151
12481993215,14cyclictest0-21swapper/122:05:151
12481993212,8cyclictest32210-21ssh23:50:401
12481993211,8cyclictest14778-21ssh00:18:481
12481993211,7cyclictest28031-21sshd19:56:201
12481993210,11cyclictest17471-21ssh23:07:111
1250399311,17cyclictest35966irq/58-eth019:50:133
12491993121,6cyclictest23492-21apt-get19:55:142
12491993116,12cyclictest0-21swapper/220:15:122
12491993115,8cyclictest9063-21cut23:20:232
12491993113,14cyclictest20485-21grep21:20:302
12491993112,11cyclictest28397-21sed20:30:272
1248199318,10cyclictest5727-21ssh22:43:111
1248199318,10cyclictest5727-21ssh22:43:101
1248199318,10cyclictest31121-21/usr/sbin/munin21:10:151
1248199316,11cyclictest10886-21ssh22:28:041
1248199316,11cyclictest10886-21ssh22:28:031
12481993117,10cyclictest16703-21ssh22:50:131
12481993116,10cyclictest0-21swapper/119:25:131
12481993111,16cyclictest12933-21ssh22:30:121
12491993018,8cyclictest9043-21mailstats20:10:292
12491993018,8cyclictest23080-21cut20:25:232
12491993015,12cyclictest2475-21ssh23:53:542
12491993011,16cyclictest32674-21/usr/sbin/munin22:05:132
12491993010,17cyclictest27015-21ls19:25:022
1248199308,9cyclictest25548-21ssh22:55:111
1248199304,11cyclictest0-21swapper/100:36:141
12481993014,12cyclictest14826-21/usr/sbin/munin23:25:151
1250399299,13cyclictest35966irq/58-eth023:50:133
1250399293,15cyclictest35966irq/58-eth021:15:133
12503992910,15cyclictest321rcuc/322:20:133
12491992917,9cyclictest2463-21tr19:30:212
12491992916,11cyclictest930-21snmpd20:09:262
12491992913,13cyclictest31684-21latency_hist19:30:022
1248199299,8cyclictest19299-21uname19:50:191
1248199297,9cyclictest31379-21rm23:33:531
1248199295,11cyclictest14532-21rm23:41:051
12481992919,4cyclictest31889-21apt-get19:30:111
1247299293,19cyclictest151rcuc/022:45:140
1247299292,19cyclictest11180-21ssh21:35:140
1250399281,12cyclictest35966irq/58-eth023:45:123
1250399281,12cyclictest35966irq/58-eth023:45:113
12491992814,11cyclictest12453-21cyclictest19:10:022
12491992814,11cyclictest12453-21cyclictest19:10:012
1248199286,7cyclictest0-21swapper/122:04:491
12481992821,5cyclictest5467-21apt-get20:10:121
12481992820,5cyclictest29087-21apt-get21:45:141
12481992818,8cyclictest0-21swapper/120:50:141
1247299285,13cyclictest0-21swapper/023:40:110
12472992817,7cyclictest16213-21sh00:20:140
1250399273,14cyclictest35966irq/58-eth022:30:143
1250399272,16cyclictest35966irq/58-eth021:10:143
1250399272,16cyclictest0-21swapper/319:15:113
12503992716,8cyclictest35966irq/58-eth022:35:243
12503992716,8cyclictest35966irq/58-eth022:35:233
1248199274,12cyclictest14144-21cat19:45:191
12481992715,9cyclictest2885-21grep20:40:211
12481992715,9cyclictest22239-21cut21:40:221
12481992713,11cyclictest512-21ssh22:21:291
12481992713,10cyclictest11344-21ntpq22:45:271
1250399263,13cyclictest35966irq/58-eth023:40:113
1250399261,19cyclictest321rcuc/321:55:133
12481992613,9cyclictest0-21swapper/119:40:101
1247299268,6cyclictest131rcu_preempt23:10:140
1247299261,19cyclictest1705-21rm00:30:140
1250399251,12cyclictest35966irq/58-eth023:05:143
1248199258,13cyclictest10004-21kthreadcore20:45:211
12481992513,9cyclictest12495-21sed19:10:011
12481992513,9cyclictest12495-21sed19:10:011
1247299253,14cyclictest20086-21ssh21:40:130
1247299252,19cyclictest25634-21ssh22:55:140
1247299252,19cyclictest10274-21apt-get20:15:130
12472992512,9cyclictest5997-21df23:20:150
1250399244,13cyclictest0-21swapper/321:30:133
1250399243,10cyclictest0-21swapper/300:35:393
1248199249,11cyclictest31902-21mailstats20:00:281
1248199246,9cyclictest23560-21sed20:25:281
12481992413,8cyclictest25890-21xargs19:20:271
12481992411,9cyclictest29383-21cat21:05:221
1247299242,7cyclictest30030-21users00:25:320
12481992314,7cyclictest0-21swapper/120:55:131
1247299231,4cyclictest0-21swapper/021:45:140
12472992311,3cyclictest131rcu_preempt00:35:190
12503992214,5cyclictest35966irq/58-eth000:10:133
1247299226,9cyclictest12-21ksoftirqd/019:10:130
1247299221,18cyclictest22001-21cpuspeed22:35:130
1247299221,18cyclictest22001-21cpuspeed22:35:120
1247299221,15cyclictest15219-21cut20:20:160
1250399212,14cyclictest35966irq/58-eth022:00:123
1247299212,15cyclictest31305-21ssh21:10:170
1247299211,16cyclictest27303-21apt-get22:20:120
1247299211,13cyclictest17561-21timerandwakeup19:45:290
1247299210,17cyclictest12842-21ls19:40:270
1250399209,7cyclictest35966irq/58-eth023:30:153
1250399202,11cyclictest0-21swapper/320:45:133
1250399202,11cyclictest0-21swapper/320:35:043
1247299209,8cyclictest452-21ntp_kernel_err23:15:260
1247299204,9cyclictest0-21swapper/021:00:140
1247299202,8cyclictest32693-21apt22:05:140
12472992012,4cyclictest16696-21cut21:55:220
1247299201,15cyclictest7575-21missed_timers19:35:230
1247299201,14cyclictest5309-21ssh23:00:210
1247299200,14cyclictest6718-21apt-get00:15:040
1250399196,9cyclictest35966irq/58-eth019:10:133
1250399193,11cyclictest0-21swapper/300:33:403
1250399192,11cyclictest0-21swapper/323:10:153
1250399192,10cyclictest0-21swapper/319:35:143
12503991911,6cyclictest35966irq/58-eth021:25:143
1250399191,13cyclictest35966irq/58-eth022:10:143
1250399191,13cyclictest0-21swapper/321:00:163
1250399191,13cyclictest0-21swapper/320:55:173
12503991910,6cyclictest35966irq/58-eth000:15:153
1247299197,8cyclictest12-21ksoftirqd/019:55:120
1247299196,8cyclictest0-21swapper/019:15:120
1247299196,3cyclictest131rcu_preempt21:15:220
1247299192,9cyclictest2190-21ssh21:30:120
1247299192,13cyclictest9278-21ssh22:10:140
12472991911,6cyclictest20943-21cron21:00:010
1247299191,12cyclictest21279-21latency_hist00:05:020
1247299190,8cyclictest18360-21fschecks_count23:45:160
1247299190,8cyclictest18360-21fschecks_count23:45:150
1250399189,6cyclictest0-21swapper/322:51:203
1250399182,10cyclictest35966irq/58-eth023:20:153
1250399182,10cyclictest0-21swapper/320:20:143
1250399182,10cyclictest0-21swapper/319:45:153
1250399182,10cyclictest0-21swapper/319:30:133
12503991812,4cyclictest0-21swapper/323:55:173
12503991811,5cyclictest0-21swapper/322:15:153
1250399181,13cyclictest0-21swapper/320:00:023
12503991810,5cyclictest35966irq/58-eth000:20:143
12503991810,5cyclictest0-21swapper/322:25:283
12503991810,5cyclictest0-21swapper/322:25:273
1247299183,13cyclictest887-21kthreadcore00:10:220
1247299181,9cyclictest25650-21cpuspeed21:25:150
1247299181,6cyclictest26070-21kthreadcore20:30:190
1247299181,13cyclictest20004-21grep20:25:170
1247299181,13cyclictest16528-21ssh21:20:130
12472991810,5cyclictest0-21swapper/020:00:170
1250399179,6cyclictest0-21swapper/321:50:053
1250399179,6cyclictest0-21swapper/300:03:063
1250399179,3cyclictest0-21swapper/323:16:503
1250399172,10cyclictest0-21swapper/320:30:133
1250399172,10cyclictest0-21swapper/320:25:153
1250399171,12cyclictest0-21swapper/322:55:203
1250399171,12cyclictest0-21swapper/319:20:133
1247299175,9cyclictest27260-21cut19:25:150
1247299174,10cyclictest21372-21kthreadcore19:50:220
1247299172,11cyclictest13051-21diskstats22:30:150
1247299171,9cyclictest2162-21fschecks_time20:40:160
1247299171,5cyclictest30598-21munin-run21:10:010
1247299171,13cyclictest21774-21df00:05:160
1247299171,13cyclictest1106-21irqcore20:05:160
1247299171,12cyclictest29675-21diskstats20:35:170
1247299171,12cyclictest14849-21apt-get23:25:150
1250399162,9cyclictest0-21swapper/320:40:153
1250399161,12cyclictest0-21swapper/321:05:283
1247299167,7cyclictest12-21ksoftirqd/020:55:010
1247299163,10cyclictest22821-21sed19:20:190
1247299162,11cyclictest6652-21kthreadcore20:10:210
1250399151,12cyclictest0-21swapper/319:25:283
1247299152,11cyclictest7433-21kernelversion20:45:190
1250399149,3cyclictest0-21swapper/320:05:243
1250399141,9cyclictest0-21swapper/320:10:143
1250399141,10cyclictest0-21swapper/320:00:213
1250399138,3cyclictest0-21swapper/320:51:303
1247299131,9cyclictest12354-21latency_hist19:10:020
1247299131,9cyclictest12354-21latency_hist19:10:010
1250399122,6cyclictest0-21swapper/319:40:123
1250399121,9cyclictest0-21swapper/320:15:153
125039991,6cyclictest0-21swapper/319:10:023
125039991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional