You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri Mar 13, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2616154212158,54sleep30-21swapper/319:09:553
261597428555,12sleep10-21swapper/119:07:361
261581828151,10sleep20-21swapper/22
261610827854,6sleep00-21swapper/00
261628399271,3cyclictest0-21swapper/33
261628399241,13cyclictest0-21swapper/321:10:123
2616283992411,10cyclictest66950irq/132-enp1s0-TxRx-300:32:143
27951002221,11sleep02795167-21modprobe0
261628399221,18cyclictest0-21swapper/321:38:343
261628399220,18cyclictest0-21swapper/321:53:223
28432032211,5sleep12843237-21munin-run22:05:001
261628399217,10cyclictest3062922-21turbostat3
2616283992113,4cyclictest29220612sleep33
2616283992111,6cyclictest2868334-21taskset3
261628399211,14cyclictest0-21swapper/323:55:113
261628399211,13cyclictest0-21swapper/33
29702492201,6sleep02970390-21ntpq0
27732892202,4sleep12746647-21diskmemload21:25:121
261628399209,8cyclictest0-21swapper/321:55:433
261628399209,3cyclictest0-21swapper/322:07:343
261628399202,14cyclictest3097508-21taskset3
2616283992015,3cyclictest0-21swapper/323:20:133
2616283992014,4cyclictest0-21swapper/319:50:173
261628399201,3cyclictest0-21swapper/33
261628399201,2cyclictest0-21swapper/323:51:583
2616283992012,5cyclictest0-21swapper/33
261628399201,15cyclictest66950irq/132-enp1s0-TxRx-300:11:273
261628399201,15cyclictest0-21swapper/322:01:533
261628399201,11cyclictest0-21swapper/33
261628399200,16cyclictest0-21swapper/33
261628399200,15cyclictest0-21swapper/323:45:423
261628399200,12cyclictest0-21swapper/300:35:213
29182592191,2sleep10-21swapper/122:45:251
29043972191,1sleep20-21swapper/222:39:502
261628399195,12cyclictest66950irq/132-enp1s0-TxRx-33
261628399195,11cyclictest0-21swapper/33
261628399194,13cyclictest66950irq/132-enp1s0-TxRx-321:30:083
261628399192,13cyclictest0-21swapper/322:55:173
2616283991916,1cyclictest0-21swapper/319:25:193
2616283991915,3cyclictest0-21swapper/323:40:003
261628399191,3cyclictest0-21swapper/33
261628399191,3cyclictest0-21swapper/33
261628399191,3cyclictest0-21swapper/323:16:583
261628399191,3cyclictest0-21swapper/322:50:043
261628399191,15cyclictest0-21swapper/33
261628399191,15cyclictest0-21swapper/323:00:333
261628399191,15cyclictest0-21swapper/320:30:033
261628399191,14cyclictest2983618-21chrt3
261628399191,14cyclictest0-21swapper/322:10:103
261628399190,2cyclictest0-21swapper/33
261628399190,2cyclictest0-21swapper/322:22:093
261628399190,16cyclictest0-21swapper/300:21:263
261628399190,15cyclictest0-21swapper/320:59:043
261628399190,14cyclictest0-21swapper/319:38:013
30978232181,4sleep03097836-21ssh0
261628399187,8cyclictest2735985-21turbostat21:04:563
261628399182,12cyclictest2621561-21turbostat19:19:543
2616283991814,2cyclictest0-21swapper/33
261628399181,3cyclictest0-21swapper/33
261628399181,3cyclictest0-21swapper/300:15:233
2616283991812,4cyclictest0-21swapper/323:05:093
261628399181,13cyclictest0-21swapper/319:10:363
2616283991810,5cyclictest2757165-21chrt21:15:143
261628399180,16cyclictest0-21swapper/320:19:003
31129992171,3sleep00-21swapper/000:36:440
30055682171,12sleep23001247-21/usr/sbin/munin2
29988392176,2sleep1171rcu_preempt23:32:201
29188452171,4sleep22746647-21diskmemload22:45:462
28078112171,2sleep00-21swapper/021:44:560
261628399171,6cyclictest2725353-21turbostat20:50:013
261628399171,6cyclictest2714676-21turbostat20:40:013
261628399171,6cyclictest2688082-21turbostat3
261628399171,13cyclictest0-21swapper/319:30:193
261628399170,13cyclictest0-21swapper/33
261628399170,13cyclictest0-21swapper/33
261628399170,13cyclictest0-21swapper/33
30533302161,2sleep00-21swapper/000:03:570
30249362161,3sleep03024948-21ssh23:47:010
30166882160,1sleep00-21swapper/023:42:410
28430362160,7sleep02843050-21ssh0
27626102161,4sleep12746647-21diskmemload21:18:031
27532732161,3sleep00-21swapper/00
261628399161,13cyclictest0-21swapper/319:45:103
261628399161,12cyclictest0-21swapper/33
261628399160,9cyclictest0-21swapper/320:09:233
261628399160,13cyclictest0-21swapper/33
261628399160,13cyclictest0-21swapper/319:40:103
30982992151,9sleep22746647-21diskmemload00:30:102
30172702151,9sleep12746647-21diskmemload23:43:271
29825552151,3sleep12746647-21diskmemload1
29745562151,9sleep22746647-21diskmemload2
29529612151,9sleep02953021-21ntpq23:05:210
29471202151,6sleep12947135-21ssh23:03:251
28869832151,2sleep00-21swapper/022:29:540
27535912151,3sleep12746647-21diskmemload1
261628399151,12cyclictest0-21swapper/320:00:153
261628399150,4cyclictest66950irq/132-enp1s0-TxRx-33
30837512141,7sleep03084835-21memory00:20:180
30516862141,3sleep22746647-21diskmemload00:01:472
30344962140,11sleep13034507-21sh1
29998922141,5sleep02999907-21ssh23:33:430
29558632141,2sleep20-21swapper/223:08:232
29374812141,8sleep00-21swapper/022:57:150
29032572141,9sleep02903267-21ssh22:38:180
28778662140,5sleep12877881-21ssh22:24:241
28778662140,5sleep12877881-21ssh1
28434872141,6sleep02843500-21rm22:05:040
28055942141,8sleep12746647-21diskmemload21:42:001
31062782130,8sleep12746647-21diskmemload00:34:261
30633302131,6sleep23063597-21df00:10:122
30608172131,6sleep12746647-21diskmemload1
30408542132,6sleep13040921-21modprobe23:55:211
30351382131,3sleep03035152-21ssh23:53:350
30093532131,6sleep12746647-21diskmemload1
29913132130,4sleep00-21swapper/023:28:570
29883412131,8sleep22988560-21ssh2
29477632130,7sleep00-21swapper/00
28699422131,2sleep00-21swapper/022:20:110
28699422131,2sleep00-21swapper/022:20:100
28607032131,8sleep02860714-21ssh22:14:530
28526892131,6sleep12852903-21hddtemp_smartct22:10:131
28141842131,3sleep22814195-21ssh2
27888572130,9sleep12788869-21sh1
27786442137,2sleep02746647-21diskmemload21:26:030
27695532137,3sleep20-21swapper/221:20:352
27519762130,1sleep20-21swapper/221:10:332
26510352131,5sleep20-21swapper/219:40:052
26204532131,7sleep20-21swapper/22
261628399130,10cyclictest0-21swapper/319:55:163
31071372121,5sleep13107402-21df1
31067132121,7sleep03106742-21idleruntime-cro00:35:010
30779872120,4sleep00-21swapper/000:16:480
30515652120,1sleep10-21swapper/100:01:381
30230582121,5sleep20-21swapper/22
30153462121,6sleep22746647-21diskmemload23:40:552
30091372121,7sleep03009149-21ssh0
29213602120,1sleep00-21swapper/00
29122202121,6sleep12746647-21diskmemload1
28565322121,5sleep22856584-21ntpq2
28334642121,6sleep00-21swapper/021:58:470
28106202122,5sleep12812355-21cut21:45:161
27790802121,6sleep22746647-21diskmemload21:26:382
27703372121,4sleep00-21swapper/021:21:380
27406212121,3sleep02740625-21mailstats0
27190102121,5sleep12719059-21ntpq20:40:181
26513332121,3sleep02651514-21expr0
26165212121,7sleep02616740-21sh19:10:120
31129162111,5sleep22746647-21diskmemload00:36:382
30857742111,4sleep12746647-21diskmemload1
30774352110,4sleep10-21swapper/100:16:051
30689062111,4sleep00-21swapper/00
30594942111,4sleep00-21swapper/000:05:300
30239292111,6sleep12746647-21diskmemload23:45:381
29919922110,4sleep10-21swapper/11
29795372111,6sleep22975012-21/usr/sbin/munin2
29722062111,3sleep12746647-21diskmemload23:16:531
29646922111,3sleep00-21swapper/023:13:300
29631462111,3sleep10-21swapper/123:11:271
29536632111,4sleep12746647-21diskmemload23:05:301
29454622110,7sleep22746647-21diskmemload23:01:142
29378032111,4sleep12746647-21diskmemload22:57:401
29285592110,2sleep10-21swapper/11
29111932111,4sleep22746647-21diskmemload22:42:132
29111232111,6sleep02911133-21ssh0
28945062111,3sleep00-21swapper/022:33:170
28939222111,4sleep12746647-21diskmemload22:32:301
28843062111,4sleep10-21swapper/122:26:221
28829892111,5sleep22883059-21ntp_states22:25:232
28617102111,3sleep00-21swapper/00
28492002111,3sleep10-21swapper/122:06:211
28403372111,3sleep20-21swapper/222:01:162
28303272111,5sleep12830388-21ntp_states21:55:221
28244912111,4sleep22746647-21diskmemload2
28215292111,5sleep12821589-21ntpq21:50:221
27958522113,1sleep20-21swapper/221:35:352
27611022111,4sleep22746647-21diskmemload21:16:032
27600062111,6sleep02760073-21/usr/sbin/munin0
27415932111,5sleep22741845-21df21:05:122
27096392111,4sleep12709881-21df_inode20:35:141
26776822111,4sleep12677700-21apt-get20:05:121
26768342111,5sleep02677051-21users20:00:260
26711332111,5sleep22671189-21ntp_states19:55:232
26566742111,5sleep22656883-21hddtemp_smartct19:45:132
30974972101,2sleep20-21swapper/22
30946512101,3sleep12746647-21diskmemload1
30864872101,3sleep20-21swapper/200:21:272
30798362101,4sleep22746647-21diskmemload00:19:152
30632942101,4sleep13063318-21apt-get00:10:111
30418262101,2sleep00-21swapper/023:55:480
29995662100,1sleep20-21swapper/223:33:162
29803942101,3sleep00-21swapper/023:21:080
29645102101,3sleep20-21swapper/22
29312252101,4sleep22931410-21expr22:55:122
29307562101,3sleep20-21swapper/222:54:572
29279382101,3sleep00-21swapper/022:51:140
29029192101,6sleep12902931-21rm22:37:521
28949472101,2sleep20-21swapper/22
28702332101,5sleep22870355-21fschecks_time2
28702332101,5sleep22870355-21fschecks_time2
28690612101,3sleep12746647-21diskmemload22:19:241
28672052100,1sleep20-21swapper/222:16:572
28313142101,4sleep20-21swapper/221:55:552
28235512101,3sleep00-21swapper/021:52:120
28140592101,3sleep00-21swapper/021:46:150
27985022101,3sleep12746647-21diskmemload21:39:061
27864192101,5sleep22786548-21/usr/sbin/munin21:30:222
27862912101,4sleep02786351-21ntpq21:30:210
27697352101,4sleep12746647-21diskmemload21:20:501
27154452101,4sleep02715516-21irqstats0
27028042101,6sleep02702938-21tail20:25:180
26939732101,5sleep12694113-21head1
26729252101,5sleep12674228-21cut20:00:151
26564072101,5sleep02656469-21cpuspeed_turbos19:45:110
26454282100,4sleep10-21swapper/11
26258232105,2sleep00-21swapper/019:15:180
26223662101,5sleep12621560-21apt-get1
3059767291,3sleep22746647-21diskmemload2
3036896291,3sleep22746647-21diskmemload23:55:132
3027877291,4sleep23028144-21df2
2851147290,2sleep20-21swapper/222:08:552
2741896293,3sleep02742104-21/usr/sbin/munin21:05:150
2741571290,3sleep10-21swapper/11
2736250290,3sleep10-21swapper/121:00:071
2728306290,1sleep20-21swapper/220:50:182
2720295291,5sleep12720384-21cpuspeed_turbos20:45:111
2708342291,4sleep12708397-21ntpq20:30:201
2708342291,4sleep12708397-21ntpq1
2693403291,4sleep2255-21systemd-journal2
2692432290,4sleep02692486-21ntpq0
2688621291,4sleep12688785-21gltestperf1
2674230291,4sleep22676050-21kthreadcore20:00:162
2658364291,4sleep12660135-21grep19:45:161
2650123290,1sleep20-21swapper/219:35:272
2641649291,3sleep22643405-21grep2
2635363291,4sleep12636328-21cut19:25:171
2634537291,4sleep11-21systemd1
2630368291,4sleep02632064-21tr19:20:140
2629191291,4sleep22629262-21munin-run19:20:012
2804495280,2sleep20-21swapper/22
2736257280,2sleep20-21swapper/221:00:122
2735971280,1sleep00-21swapper/020:59:570
2716790281,3sleep20-21swapper/220:40:152
2709614280,4sleep02709638-21grep20:35:120
2699011281,3sleep22699200-21cstates20:25:122
2694956281,3sleep02696689-21grep20:20:180
2687546280,1sleep10-21swapper/120:10:241
2683005280,3sleep22683024-21apt-get2
2671715280,1sleep10-21swapper/119:55:301
2671027281,3sleep02671078-21ntpq19:55:220
2663093281,3sleep02664858-21kthreadcore0
2636329281,3sleep22638125-21kthreadcore2
2735634270,1sleep10-21swapper/120:55:451
2726615271,3sleep10-21swapper/120:50:171
2726148272,2sleep02726612-21kthreadcore0
2714446270,1sleep20-21swapper/220:37:162
2708288271,2sleep00-21swapper/020:30:200
2708288271,2sleep00-21swapper/00
2706248271,2sleep20-21swapper/220:30:182
2706248271,2sleep20-21swapper/22
2698989270,2sleep10-21swapper/11
2687761270,1sleep00-21swapper/020:11:040
2666681270,1sleep20-21swapper/22
2650349270,1sleep00-21swapper/00
2645004270,1sleep00-21swapper/019:32:450
2616741271,3sleep10-21swapper/119:10:131
2721844261,2sleep20-21swapper/220:45:162
2720273261,2sleep00-21swapper/00
2698450260,1sleep20-21swapper/220:21:282
2679132260,2sleep20-21swapper/220:05:172
2666503260,1sleep10-21swapper/119:51:491
2645068260,4sleep10-21swapper/11
2639853260,3sleep02639892-21awk19:30:000
2681648250,1sleep00-21swapper/00
2651515250,1sleep10-21swapper/11
2634496250,1sleep20-21swapper/22
2731203240,1sleep20-21swapper/220:55:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional