You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 13:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Aug 07, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
140099580,30cyclictest12571-21ssh23:30:120
1423995216,26cyclictest27-21ksoftirqd/222:30:132
141299461,40cyclictest4344-21apt-get20:20:131
1423994526,8cyclictest0-21swapper/221:35:122
142399451,19cyclictest0-21swapper/220:45:142
141299441,38cyclictest15967-21apt-get19:25:131
1423994331,7cyclictest7149-21apt-get23:45:132
141299432,36cyclictest11185-21apt-get19:20:141
1423994221,17cyclictest0-21swapper/222:05:112
1423994118,12cyclictest0-21swapper/219:20:132
141299411,37cyclictest13964-21apt-get20:30:131
1423994012,12cyclictest27-21ksoftirqd/200:28:012
141299402,32cyclictest12545-21apt-get23:30:131
1423993922,8cyclictest0-21swapper/223:30:122
1423993916,19cyclictest16503-21apt-get22:20:132
141299391,26cyclictest14557-21ssh22:55:121
141299390,26cyclictest13088-21sh23:46:491
142399388,21cyclictest18793-21ssh00:09:272
1423993821,13cyclictest0-21swapper/220:20:112
141299381,31cyclictest25669-21apt-get19:35:131
141299381,31cyclictest25669-21apt-get19:35:131
141299381,22cyclictest5218-21ssh23:43:081
1435993716,9cyclictest35966irq/58-eth000:25:133
1423993713,18cyclictest0-21swapper/200:30:152
141299371,33cyclictest12974-21apt-get22:00:111
141299370,32cyclictest9180-21apt-get20:25:111
140099374,29cyclictest16132-21ssh23:50:140
140099373,14cyclictest23599-21ssh23:00:130
143599363,19cyclictest35966irq/58-eth020:15:123
1423993619,13cyclictest3585-21apt-get23:25:122
141299361,21cyclictest11052-21/usr/sbin/munin22:35:261
141299361,19cyclictest19003-21ssh00:09:411
140099362,9cyclictest631-21Async09:36:310
142399356,12cyclictest6612-21ssh00:00:312
1423993524,7cyclictest28601-21ssh00:15:152
1423993514,11cyclictest0-21swapper/219:25:132
1423993513,8cyclictest18770-21ls21:25:292
1423993511,22cyclictest135-21systemd-journal23:02:402
141299351,31cyclictest2864-21apt-get19:45:111
141299351,30cyclictest27376-21missed_timers21:30:241
140099352,9cyclictest1379-21kworker/u8:000:16:440
1423993422,8cyclictest19831-21diskmemload22:35:122
1423993419,11cyclictest12980-21ssh22:00:112
141299346,13cyclictest0-21swapper/121:16:161
141299341,28cyclictest28592-21apt-get00:15:141
141299341,21cyclictest26161-21ssh21:49:011
141299341,19cyclictest11463-21ssh21:21:241
141299341,17cyclictest27321-21ssh23:36:351
141299341,17cyclictest0-21swapper/122:27:281
141299340,29cyclictest30928-21apt-get22:10:131
141299340,22cyclictest17084-21/usr/sbin/munin19:55:311
140099345,4cyclictest131rcu_preempt22:35:130
143599331,19cyclictest35966irq/58-eth023:10:113
142399337,13cyclictest23155-21sh23:53:092
142399331,24cyclictest20238-21ssh21:10:142
141299332,26cyclictest0-21swapper/121:10:131
141299332,25cyclictest27055-21apt-get23:20:151
141299331,29cyclictest28396-21apt-get20:45:141
141299331,20cyclictest26302-21rm22:42:121
141299331,20cyclictest23010-21sh21:29:271
141299331,17cyclictest19215-21ssh00:25:491
141299331,17cyclictest10896-21users19:15:311
141299331,17cyclictest10667-21ssh22:34:591
141299330,27cyclictest385-21apt-get21:35:121
140099332,27cyclictest3607-21ssh23:25:130
1435993223,5cyclictest35966irq/58-eth021:45:153
142399327,10cyclictest0-21swapper/221:45:162
142399326,12cyclictest0-21swapper/222:50:102
1423993224,4cyclictest20127-21apt-get22:40:152
1423993213,14cyclictest0-21swapper/223:55:132
141299322,27cyclictest0-21swapper/122:20:131
141299321,26cyclictest0-21swapper/122:50:141
141299321,18cyclictest0-21swapper/121:44:131
141299321,17cyclictest9082-21ssh21:55:451
140099322,10cyclictest0-21swapper/023:40:140
140099321,27cyclictest32221-21ssh00:35:150
143599312,17cyclictest35966irq/58-eth000:00:143
142399317,9cyclictest30910-21ssh22:26:112
1423993126,2cyclictest23869-21apt-get21:30:132
1423993124,3cyclictest30601-21apt-get23:40:142
142399312,24cyclictest12002-21ssh00:23:002
1423993114,8cyclictest9421-21/usr/sbin/munin21:40:132
1423993111,16cyclictest11269-21proc_pri19:50:262
1423993110,7cyclictest5287-21ssh22:50:012
141299311,27cyclictest5010-21ntp_states19:10:251
141299311,27cyclictest10637-21sh00:21:201
141299311,21cyclictest0-21swapper/122:15:141
141299311,17cyclictest0-21swapper/123:52:491
141299311,17cyclictest0-21swapper/123:05:461
141299310,28cyclictest10373-21apt-get21:00:131
141299310,28cyclictest10098-21sshd20:55:431
141299310,27cyclictest31956-21apt-get20:15:121
141299310,27cyclictest26812-21apt-get20:10:031
140099312,23cyclictest12-21ksoftirqd/023:10:120
143599303,12cyclictest35966irq/58-eth023:25:143
142399302,21cyclictest0-21swapper/220:00:042
1423993013,14cyclictest18655-21ntp_192.168.11421:05:242
1423993013,13cyclictest30325-21latency_hist19:40:012
1423993013,13cyclictest30325-21latency_hist19:40:002
1423993010,17cyclictest28742-21cut21:50:222
1423993010,17cyclictest28742-21cut21:50:222
141299301,27cyclictest30477-21ssh23:56:061
141299301,26cyclictest9817-21irqstats23:10:191
141299301,26cyclictest879-21awk20:50:131
141299301,17cyclictest7798-21sh00:02:001
141299301,16cyclictest5411-21sh00:36:281
141299300,26cyclictest14927-21apt-get21:05:031
140099306,21cyclictest12-21ksoftirqd/022:50:150
143599293,16cyclictest35966irq/58-eth021:10:123
143599292,18cyclictest35966irq/58-eth022:05:133
143599292,12cyclictest35966irq/58-eth023:45:143
142399299,7cyclictest93066snmpd20:31:002
142399298,8cyclictest29015-21ssh23:38:372
142399296,10cyclictest0-21swapper/222:16:102
1423992916,10cyclictest0-21swapper/220:25:112
1423992914,12cyclictest4203-21/usr/sbin/munin20:50:242
1423992914,12cyclictest32721-21/usr/sbin/munin21:15:222
1423992913,13cyclictest30637-21/usr/sbin/munin20:10:332
1423992913,13cyclictest10364-21find21:00:132
1423992912,8cyclictest5079-21processes19:10:252
1423992912,14cyclictest32386-21tune2fs00:35:172
1423992912,13cyclictest10182-21tail19:15:292
1423992911,9cyclictest7877-21mailstats21:55:282
141299296,20cyclictest23608-21/usr/sbin/munin20:40:221
141299291,25cyclictest25930-21wc22:05:301
141299291,23cyclictest0-21swapper/120:05:121
141299290,27cyclictest23137-21ssh00:30:131
141299290,16cyclictest1020-21ssh21:51:541
141299290,16cyclictest1020-21ssh21:51:531
140099294,7cyclictest131rcu_preempt20:45:130
140099293,4cyclictest131rcu_preempt19:35:130
140099293,4cyclictest131rcu_preempt19:35:120
140099293,22cyclictest20159-21ssh22:40:150
140099291,24cyclictest23891-21ssh21:30:130
142399287,18cyclictest2843-21kthreadcore23:05:232
1423992815,6cyclictest27058-21ntp_states20:40:242
1423992813,12cyclictest31005-21/usr/sbin/munin23:20:292
1423992812,9cyclictest13638-21cat23:10:312
1423992812,13cyclictest2839-21latency_hist19:45:012
1423992812,13cyclictest24212-21cat00:10:292
1423992812,13cyclictest19302-21sed20:35:202
1423992812,13cyclictest18262-21cut23:15:182
1423992811,8cyclictest14933-21tail22:55:182
1423992811,13cyclictest25602-21date19:35:012
141299281,24cyclictest26635-21latency_hist23:20:011
141299281,24cyclictest23549-21/usr/sbin/munin23:00:141
141299281,24cyclictest20376-21uname00:10:181
141299281,23cyclictest7444-21apt-get19:50:051
141299281,21cyclictest0-21swapper/119:30:121
141299280,26cyclictest10135-21ssh23:27:321
141299280,25cyclictest30496-21apt-get19:40:141
141299280,25cyclictest29001-21acpi22:45:131
141299280,25cyclictest18780-21munin-node20:35:131
140099282,22cyclictest1761-21ssh00:00:130
143599273,12cyclictest35966irq/58-eth023:05:143
1435992710,12cyclictest35966irq/58-eth023:15:173
142399277,12cyclictest13281-21tr19:55:202
142399271,21cyclictest5939-21ssh21:20:182
1435992614,8cyclictest35966irq/58-eth000:30:163
142399266,16cyclictest31938-21/usr/sbin/munin20:15:292
142399265,6cyclictest0-21swapper/220:05:192
1423992614,8cyclictest31048-21diskstats22:10:152
1423992612,12cyclictest0-21swapper/219:45:122
141299261,22cyclictest18727-21cut20:00:211
140099263,18cyclictest19660-21ssh00:10:130
140099262,22cyclictest21949-21ssh22:05:110
140099262,19cyclictest416-21ssh21:35:140
1400992614,9cyclictest0-21swapper/023:05:150
143599253,14cyclictest0-21swapper/323:21:453
143599252,16cyclictest321rcuc/319:10:133
141299251,21cyclictest1467-21sed19:10:011
140099252,19cyclictest4062-21diskmemload21:55:130
140099252,19cyclictest18098-21ssh23:15:170
140099252,18cyclictest29043-21rm22:45:150
140099251,21cyclictest25503-21ssh22:25:140
143599243,13cyclictest0-21swapper/300:05:133
140099242,19cyclictest14894-21ssh21:25:150
140099242,17cyclictest16517-21ssh22:20:130
143599233,10cyclictest0-21swapper/321:35:143
1435992311,8cyclictest0-21swapper/322:20:133
140099233,16cyclictest27403-21rm21:50:140
140099233,16cyclictest27403-21rm21:50:130
140099231,18cyclictest21568-21sh23:35:130
140099231,18cyclictest18390-21ssh21:45:140
143599229,6cyclictest0-21swapper/323:50:203
143599223,13cyclictest35966irq/58-eth000:35:143
1435992215,4cyclictest35966irq/58-eth023:35:133
1435992210,9cyclictest0-21swapper/322:17:593
1423992212,7cyclictest5647-21/usr/sbin/munin20:55:132
140099227,4cyclictest131rcu_preempt21:15:190
140099222,17cyclictest19831-21diskmemload23:56:450
140099222,16cyclictest20219-21apt-get21:10:130
140099221,18cyclictest22538-21timerwakeupswit20:35:310
140099221,17cyclictest12991-21ssh22:00:120
1435992113,4cyclictest0-21swapper/322:30:163
1435992112,4cyclictest0-21swapper/321:15:233
1435992110,7cyclictest35966irq/58-eth022:00:123
140099219,9cyclictest8907-21wc00:20:290
1400992114,5cyclictest0-21swapper/022:30:110
140099211,17cyclictest30571-21ls19:40:130
1400992110,8cyclictest10679-21cat23:45:230
143599204,10cyclictest0-21swapper/322:50:133
143599203,12cyclictest0-21swapper/300:19:443
143599202,13cyclictest0-21swapper/323:56:443
1435992011,6cyclictest0-21swapper/322:10:363
143599201,15cyclictest0-21swapper/323:00:213
140099202,11cyclictest14116-21ssh00:25:140
1400992013,5cyclictest0-21swapper/020:50:180
1400992011,5cyclictest29042-21kthreadcore23:20:220
140099201,14cyclictest10131-21sendmail21:00:010
143599199,8cyclictest0-21swapper/321:56:163
143599199,8cyclictest0-21swapper/321:50:043
143599199,8cyclictest0-21swapper/321:50:033
143599199,6cyclictest35966irq/58-eth023:40:153
143599199,6cyclictest0-21swapper/321:44:113
143599199,3cyclictest0-21swapper/322:59:233
143599193,11cyclictest0-21swapper/323:31:453
143599193,11cyclictest0-21swapper/300:23:003
143599192,10cyclictest0-21swapper/319:50:053
1435991911,6cyclictest0-21swapper/322:43:473
143599191,14cyclictest0-21swapper/321:25:133
143599191,13cyclictest0-21swapper/320:45:233
143599191,13cyclictest0-21swapper/319:20:173
143599191,12cyclictest0-21swapper/321:05:283
140099199,7cyclictest10786-21users21:20:360
140099196,4cyclictest131rcu_preempt19:45:130
140099195,8cyclictest0-21swapper/019:55:130
140099193,12cyclictest9444-21ssh21:40:130
140099191,6cyclictest15965-21cat22:55:210
140099191,15cyclictest2111-21ssh22:10:230
140099191,14cyclictest17127-21apt-get20:00:050
143599189,6cyclictest35966irq/58-eth022:25:153
143599189,6cyclictest0-21swapper/321:23:173
143599189,6cyclictest0-21swapper/300:10:543
143599182,10cyclictest0-21swapper/320:10:043
143599182,10cyclictest0-21swapper/320:00:053
143599182,10cyclictest0-21swapper/319:35:053
143599182,10cyclictest0-21swapper/319:35:043
143599181,13cyclictest0-21swapper/321:00:233
140099188,7cyclictest7889-21ps22:15:180
140099188,7cyclictest15244-21apt-get21:05:130
140099183,11cyclictest31997-21ssh00:35:020
1400991811,5cyclictest26793-21sshd20:07:210
140099181,14cyclictest11406-21uname00:05:200
140099181,11cyclictest4128-21awk20:20:000
1400991810,5cyclictest28164-21idleruntime-cro20:45:010
143599179,6cyclictest0-21swapper/322:39:053
143599179,5cyclictest35966irq/58-eth021:30:133
143599172,10cyclictest0-21swapper/322:45:153
143599172,10cyclictest0-21swapper/320:40:123
143599172,10cyclictest0-21swapper/320:35:153
143599172,10cyclictest0-21swapper/320:25:123
143599172,10cyclictest0-21swapper/319:25:143
142399171,8cyclictest1386-21cyclictest19:10:022
140099179,5cyclictest12319-21kthreadcore21:00:210
140099177,8cyclictest7759-21apt-get19:50:120
140099173,10cyclictest0-21swapper/020:25:120
140099172,13cyclictest16353-21kthreadcore20:30:200
140099172,11cyclictest2048-21sed19:10:180
1400991712,3cyclictest536-21usb-storage20:10:040
1400991712,3cyclictest0-21swapper/019:15:150
1400991711,4cyclictest16173-21hwlatdetect19:25:160
1400991710,4cyclictest0-21swapper/019:32:150
143599167,5cyclictest0-21swapper/320:20:133
143599161,9cyclictest0-21swapper/319:40:163
140099162,8cyclictest0-21swapper/019:20:150
140099151,11cyclictest1410-21latency_hist19:10:020
143599149,3cyclictest0-21swapper/320:05:323
143599149,3cyclictest0-21swapper/319:56:423
143599149,3cyclictest0-21swapper/319:32:333
143599138,3cyclictest35966irq/58-eth020:50:193
143599132,9cyclictest0-21swapper/320:30:163
143599132,8cyclictest0-21swapper/321:00:023
143599131,9cyclictest0-21swapper/319:15:193
143599121,6cyclictest0-21swapper/319:45:163
143599111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional