You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 05:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat May 23, 2026 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3088599694,49cyclictest13109-21ssh00:25:121
30902996827,25cyclictest35966irq/58-eth021:40:103
30902996744,13cyclictest35966irq/58-eth022:10:123
3088199523,38cyclictest11139-21ssh23:30:130
30893994714,22cyclictest27-21ksoftirqd/223:15:112
3090299444,23cyclictest35966irq/58-eth022:25:123
3090299442,24cyclictest35966irq/58-eth021:30:133
3090299439,21cyclictest35966irq/58-eth000:30:143
3088199434,31cyclictest10373-21ssh22:00:140
3090299422,23cyclictest35966irq/58-eth023:45:133
30893994218,17cyclictest27-21ksoftirqd/200:15:132
3088199421,32cyclictest4125-21ssh00:20:120
3090299411,24cyclictest35966irq/58-eth000:20:123
3089399412,23cyclictest12135-21ssh21:25:122
3088599411,31cyclictest27003-21rm22:45:121
3089399404,26cyclictest28378-21ssh22:10:122
3088199391,30cyclictest0-21swapper/022:45:120
3090299384,23cyclictest35966irq/58-eth000:35:123
3088199382,18cyclictest10674-21ssh22:53:090
3088199382,18cyclictest10674-21ssh22:53:080
30893993728,5cyclictest27-21ksoftirqd/221:35:122
3090299362,20cyclictest35966irq/58-eth023:40:133
3090299361,13cyclictest35966irq/58-eth019:50:113
3089399368,15cyclictest32257-21ssh22:30:132
3089399364,12cyclictest261rcuc/221:20:152
3088199363,25cyclictest14085-21ssh22:20:130
3088199361,3cyclictest131rcu_preempt19:15:120
3089399353,20cyclictest7333-21ssh23:10:132
3089399342,25cyclictest24787-21ssh21:50:142
3090299331,19cyclictest35966irq/58-eth023:55:113
3089399332,28cyclictest1640-21ssh22:48:082
3089399325,17cyclictest27346irq/40-dwc2_hso19:45:112
3089399322,24cyclictest18523-21ssh00:10:142
3088199321,20cyclictest22110-21ssh00:30:120
30881993211,14cyclictest0-21swapper/021:30:130
3090299311,6cyclictest0-21swapper/323:30:123
3089399319,13cyclictest27-21ksoftirqd/220:15:132
3089399312,25cyclictest30764-21ssh23:21:092
3090299302,15cyclictest35966irq/58-eth022:30:133
3090299301,19cyclictest35966irq/58-eth019:20:133
3090299300,20cyclictest35966irq/58-eth023:00:123
3090299293,21cyclictest35966irq/58-eth021:45:153
3090299293,14cyclictest35966irq/58-eth020:30:113
3089399295,13cyclictest27346irq/40-dwc2_hso20:25:122
3089399292,18cyclictest27346irq/40-dwc2_hso22:35:122
3088199292,22cyclictest29026-21ssh22:26:410
3088199291,20cyclictest30704-21ssh23:05:120
3090299286,18cyclictest0-21swapper/322:35:123
3090299283,13cyclictest35966irq/58-eth022:50:123
3090299283,13cyclictest35966irq/58-eth022:50:113
30902992813,12cyclictest0-21swapper/323:15:123
3089399287,15cyclictest0-21swapper/222:25:122
3089399284,13cyclictest27346irq/40-dwc2_hso20:55:112
3089399283,21cyclictest27-21ksoftirqd/223:26:462
3089399283,20cyclictest0-21swapper/222:21:092
3089399283,16cyclictest27346irq/40-dwc2_hso21:05:122
3089399283,16cyclictest27-21ksoftirqd/221:10:122
3089399283,15cyclictest16928-21diskmemload21:55:142
3089399281,12cyclictest24180-21ssh22:42:092
3088599283,17cyclictest6734-21ssh21:40:111
3088199283,12cyclictest19379-21ssh22:05:120
3090299271,17cyclictest35966irq/58-eth000:00:133
3089399278,15cyclictest12725-21/usr/sbin/munin22:55:142
3089399274,14cyclictest131rcu_preempt21:00:152
3089399272,20cyclictest6203-21ssh23:09:082
3088599274,14cyclictest4996-21ssh22:15:131
30885992716,6cyclictest21-21ksoftirqd/120:25:121
3088199272,20cyclictest18012-21ssh22:56:080
3088199272,19cyclictest1-21systemd23:26:450
3090299261,5cyclictest0-21swapper/323:36:313
3090299261,4cyclictest0-21swapper/321:24:473
3090299261,3cyclictest0-21swapper/319:55:153
3090299261,17cyclictest35966irq/58-eth022:20:133
3089399262,19cyclictest0-21swapper/220:20:112
3090299253,14cyclictest0-21swapper/323:21:083
3090299252,13cyclictest35966irq/58-eth022:00:133
3090299251,9cyclictest0-21swapper/322:40:203
30902992518,4cyclictest0-21swapper/322:15:193
3090299251,5cyclictest0-21swapper/321:58:083
3090299251,3cyclictest0-21swapper/323:05:183
3090299251,3cyclictest0-21swapper/322:45:163
3090299251,3cyclictest0-21swapper/321:15:233
3090299251,3cyclictest0-21swapper/320:20:133
3089399259,13cyclictest21727-21cron00:30:022
3089399258,3cyclictest3599-21ssh22:50:112
3089399258,3cyclictest3599-21ssh22:50:112
3089399252,18cyclictest930-21snmpd19:50:132
3089399251,9cyclictest10412-21ssh22:16:082
30893992517,5cyclictest22591-21ssh21:30:202
3089399251,15cyclictest9620-21grep19:55:152
3089399251,15cyclictest24102-21fschecks_time23:55:152
3088599252,4cyclictest131rcu_preempt22:30:121
3088599252,19cyclictest16928-21diskmemload23:50:151
30885992515,6cyclictest16928-21diskmemload00:10:241
3090299241,6cyclictest0-21swapper/323:25:023
3090299241,6cyclictest0-21swapper/300:18:423
3090299241,6cyclictest0-21swapper/300:14:173
3090299241,5cyclictest0-21swapper/321:52:173
3090299241,5cyclictest0-21swapper/320:55:113
3090299241,5cyclictest0-21swapper/320:45:203
3090299241,5cyclictest0-21swapper/300:28:013
3090299241,4cyclictest0-21swapper/323:10:143
3090299241,4cyclictest0-21swapper/321:35:173
3090299241,3cyclictest0-21swapper/323:50:183
3090299241,3cyclictest0-21swapper/322:05:163
3090299241,3cyclictest0-21swapper/319:15:153
3090299241,3cyclictest0-21swapper/300:05:233
30893992413,9cyclictest8447-21ssh00:04:132
3089399241,18cyclictest9510-21apt-get00:05:132
3088199246,10cyclictest12-21ksoftirqd/022:30:140
3088199241,18cyclictest13130-21ssh00:25:130
3090299232,7cyclictest0-21swapper/322:55:173
3090299231,6cyclictest0-21swapper/320:15:293
3090299231,5cyclictest0-21swapper/321:00:223
3090299231,5cyclictest0-21swapper/320:40:263
3090299231,5cyclictest0-21swapper/320:10:283
3090299231,4cyclictest0-21swapper/321:10:433
3090299231,4cyclictest0-21swapper/319:10:183
3090299231,3cyclictest0-21swapper/320:25:293
3090299231,3cyclictest0-21swapper/320:05:183
3090299231,3cyclictest0-21swapper/320:00:153
3090299231,3cyclictest0-21swapper/319:35:263
3090299231,3cyclictest0-21swapper/319:30:153
3090299231,3cyclictest0-21swapper/319:25:153
3089399235,11cyclictest27346irq/40-dwc2_hso19:40:152
3089399233,15cyclictest18552-21ntpq23:50:272
3089399232,17cyclictest10348-21ssh21:40:232
3089399232,16cyclictest0-21swapper/223:01:452
30893992311,9cyclictest26264-21ssh21:15:032
3089399231,14cyclictest27346irq/40-dwc2_hso20:10:122
3089399231,12cyclictest29175-21/usr/sbin/munin23:40:322
3088599231,18cyclictest5612-21ssh23:25:221
30885992311,3cyclictest131rcu_preempt23:30:171
3088199231,16cyclictest733266ssh23:10:120
3090299221,5cyclictest0-21swapper/321:26:093
3090299221,4cyclictest0-21swapper/320:50:153
3090299221,4cyclictest0-21swapper/320:50:153
3090299221,4cyclictest0-21swapper/320:35:263
3090299221,4cyclictest0-21swapper/319:45:143
3090299221,3cyclictest0-21swapper/321:05:153
3090299221,3cyclictest0-21swapper/319:40:233
3089399221,9cyclictest1042666ssh23:45:302
3089399221,5cyclictest0-21swapper/220:00:122
3089399221,16cyclictest22889-21cat22:05:232
3089399221,16cyclictest0-21swapper/223:30:092
3089399221,11cyclictest25078-21latency_hist20:45:022
30893992210,4cyclictest1448-21kthreadcore00:35:202
3089399220,10cyclictest16520-21irqrtprio21:45:202
3088599229,6cyclictest21-21ksoftirqd/120:30:121
3088599222,16cyclictest29213-21ssh23:40:131
3088599221,17cyclictest0-21swapper/122:50:131
3088599221,17cyclictest0-21swapper/122:50:131
3088199223,15cyclictest513-21ssh00:00:140
30881992213,6cyclictest0-21swapper/022:15:120
3088199221,18cyclictest17929-21ssh22:40:120
30881992211,8cyclictest12109-21apt-get21:25:110
30881992211,6cyclictest0-21swapper/023:19:080
3089399213,9cyclictest11404-21kthreadcore20:30:192
3089399213,15cyclictest5466-21kthreadcore19:15:202
3089399213,15cyclictest11289-21tr22:00:192
3089399213,14cyclictest27346irq/40-dwc2_hso19:20:142
3089399212,15cyclictest25061-21tr00:30:222
3089399211,6cyclictest0-21swapper/219:35:112
3089399211,13cyclictest20166-21ssh23:35:102
3089399211,10cyclictest20325-21cat20:40:022
3088599215,10cyclictest0-21swapper/120:45:121
3088599212,14cyclictest3711-21ssh23:06:091
3088599212,10cyclictest5839-21ssh23:45:121
30885992116,3cyclictest0-21swapper/121:45:161
30885992112,7cyclictest32538-21cron00:00:011
3088599211,16cyclictest930-21snmpd21:20:131
3088199219,3cyclictest131rcu_preempt23:20:290
3088199211,17cyclictest23918-21ssh23:55:120
3088199211,13cyclictest640-21switchtime23:40:280
3089399206,5cyclictest25313-21fschecks_count20:45:152
3089399202,9cyclictest19136-21grep20:05:162
3089399201,9cyclictest5833-21kthreadcore00:20:202
3089399201,9cyclictest14691-21kthreadcore19:25:192
3089399201,9cyclictest1338-21unixbench_multi20:50:452
3089399201,9cyclictest1338-21unixbench_multi20:50:442
3089399201,8cyclictest18073-21fschecks_time19:30:172
3089399201,16cyclictest1248-21kthreadcore19:10:222
3088599207,10cyclictest614-21dbus-daemon22:11:351
3088599202,10cyclictest26409-21ssh21:15:121
30885992013,4cyclictest19959-21irqrtprio22:05:191
3088599200,15cyclictest10386-21sh22:00:141
3088199205,3cyclictest131rcu_preempt21:11:270
3088199202,13cyclictest31167-21ssh00:35:130
3088199201,16cyclictest15825-21ssh21:45:150
3088199201,14cyclictest1512-21cut21:55:160
3088199201,13cyclictest4776-21uniq19:50:150
3088199201,13cyclictest29959-21find20:50:120
3088199201,13cyclictest29959-21find20:50:120
3088199201,12cyclictest26423-21ssh21:15:120
3088199201,11cyclictest29445-21kthreadcore00:15:210
3090299191,5cyclictest0-21swapper/319:10:023
3088599199,7cyclictest11684-21cat23:10:281
3088599198,7cyclictest7375-21munin-run00:40:011
3088599194,4cyclictest4545-21irqcore00:20:181
3088599191,2cyclictest131rcu_preempt23:35:091
30885991912,4cyclictest13161-21sh00:05:241
3088599191,16cyclictest16382-21ssh23:15:121
30885991911,5cyclictest23662-21kthreadcore23:00:191
3088599191,12cyclictest9000-21ls22:35:141
30885991910,7cyclictest15470-21apt-get20:35:101
3088199192,4cyclictest12-21ksoftirqd/021:50:190
30881991913,4cyclictest0-21swapper/020:05:160
30881991910,7cyclictest23679-21cut23:00:190
3088599189,7cyclictest19884-21cut22:40:191
3088599189,5cyclictest14843-21kthreadcore22:20:201
3088599186,3cyclictest131rcu_preempt21:30:221
30885991811,4cyclictest25074-21kthreadcore00:30:221
30885991810,6cyclictest27312-21cut21:50:221
30885991810,6cyclictest20362-21sshd20:40:081
30885991810,5cyclictest25956-21irqcore23:20:171
3088199189,6cyclictest11727-21apt-config21:25:010
3088199187,8cyclictest23995-21ssh23:35:260
3088199186,9cyclictest25112-21wc20:45:020
3088199181,7cyclictest31777-21grep21:35:190
3088199181,6cyclictest16928-21diskmemload23:51:440
30881991812,4cyclictest0-21swapper/019:55:280
30881991811,4cyclictest30358-21kthreadcore22:10:210
3088199181,13cyclictest22672-21apt-get19:35:100
30881991810,5cyclictest0-21swapper/020:00:250
3088599179,6cyclictest13987-21cron20:00:011
3088599178,7cyclictest12893-21ntp_states19:55:261
3088599175,3cyclictest131rcu_preempt21:10:161
3088599173,11cyclictest12109-21apt-get21:25:121
3088599172,13cyclictest17050-21latency_hist21:10:021
3088599171,6cyclictest686-21in:imklog20:55:021
3088599171,6cyclictest686-21in:imklog20:55:011
30885991714,2cyclictest21-21ksoftirqd/100:15:121
3088599171,14cyclictest8460-21unixbench_multi19:50:281
3088599171,13cyclictest8382-21diskstats19:20:151
3088599171,13cyclictest3113-21kthreadcore20:55:171
3088599171,13cyclictest30864-21irqstats21:35:191
3088599171,13cyclictest1334-21ssh21:55:121
3088199179,6cyclictest5531-21vmstat20:20:280
3088199171,6cyclictest8890-21apt22:35:120
3088199171,6cyclictest16928-21diskmemload23:46:560
3088199171,13cyclictest13631-21grep00:05:290
30881991710,5cyclictest31389-21cut19:10:180
3088199170,12cyclictest15455-21find20:35:090
3088599169,4cyclictest23382-21fschecks_time22:25:141
3088599167,6cyclictest7956-21munin-run19:20:021
3088599165,3cyclictest0-21swapper/120:05:301
3088599163,11cyclictest31604-21irqrtprio19:10:181
3088599162,9cyclictest464-21find00:00:121
3088599162,9cyclictest0-21swapper/119:30:131
3088599162,11cyclictest1644-21kthreadcore19:45:201
3088599162,11cyclictest12587-21ssh22:55:121
3088599161,13cyclictest4449-21ntpq20:20:221
3088599161,12cyclictest7166-21apt-get21:00:131
3088599161,11cyclictest24052-21hwlatdetect20:10:161
30885991610,4cyclictest0-21swapper/119:40:161
3088199169,5cyclictest30611-21cut19:40:220
3088199169,5cyclictest28234-21cut20:45:220
3088199168,6cyclictest6663-21sshd21:40:080
3088199168,6cyclictest15003-21kthreadcore19:25:190
3088199168,6cyclictest12001-21sshd21:00:550
3088199162,11cyclictest20290-21kthreadcore00:10:200
3088199162,10cyclictest0-21swapper/019:30:120
3088199161,8cyclictest32687-21irqcore19:45:160
3088199161,8cyclictest28673-21cpuspeed20:15:130
3088199161,11cyclictest0-21swapper/021:05:130
3088599153,9cyclictest29556-21kthreadcore20:15:181
3088599152,10cyclictest15002-21kthreadcore19:25:201
3088599151,11cyclictest0-21swapper/119:35:111
3088199158,5cyclictest4015-21tr20:55:180
3088199151,12cyclictest25315-21cut20:10:190
3088199151,11cyclictest8288-21apt-get19:20:140
3088199151,11cyclictest30942-21latency_hist19:10:020
3088199151,11cyclictest13645-21kthreadcore20:30:210
3088199149,3cyclictest536-21usb-storage20:25:050
3089399134,6cyclictest30910-21latency_hist19:10:022
3088599131,9cyclictest30774-21latency_hist19:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional