You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 03:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Dec 07, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3936680211959,56sleep30-21swapper/319:07:523
393667328252,10sleep10-21swapper/119:07:451
393675728054,21sleep20-21swapper/219:08:512
393678027454,14sleep00-21swapper/019:09:090
393697899262,20cyclictest138649-21taskset23:59:083
3936978992521,2cyclictest0-21swapper/323:40:003
393697899251,19cyclictest191869-21taskset00:32:433
393697899251,12cyclictest147200-21turbostat00:05:013
3936978992421,1cyclictest0-21swapper/319:30:003
3936978992418,3cyclictest0-21swapper/322:35:153
3936978992411,9cyclictest4135346-21taskset21:52:363
393697899238,9cyclictest4082474-21taskset21:20:013
3936978992318,2cyclictest0-21swapper/323:34:593
3936978992311,8cyclictest4157505-21chrt22:05:393
393697899227,6cyclictest51623-21taskset23:03:563
393697899222,15cyclictest178294-21turbostat00:25:003
393697899222,14cyclictest52286-21sleep23:09:553
393697899222,13cyclictest2315-21taskset22:30:323
3936978992218,2cyclictest0-21swapper/300:13:273
3936978992217,3cyclictest0-21swapper/322:50:203
393697899212,13cyclictest4081483-21taskset21:18:493
393697899212,12cyclictest4144782-21taskset22:00:063
393697899212,12cyclictest27705-21taskset22:48:163
3936978992119,1cyclictest0-21swapper/319:45:013
3936978992118,1cyclictest0-21swapper/319:50:133
3936978992116,3cyclictest0-21swapper/321:45:183
3936978992113,5cyclictest66550irq/131-enp1s0-TxRx-321:10:243
41842042201,14sleep14185033-21cut22:25:181
41753512200,2sleep20-21swapper/222:19:462
393697899205,12cyclictest4104540-21taskset21:33:123
393697899203,13cyclictest0-21swapper/300:17:493
393697899202,13cyclictest4097464-21taskset21:29:193
393697899202,12cyclictest4175789-21taskset22:20:023
393697899201,15cyclictest4066342-21turbostat21:10:003
393697899201,15cyclictest4023693-21taskset20:28:383
393697899201,14cyclictest123516-21chrt23:50:073
393697899192,12cyclictest4174032-21taskset22:17:213
393697899192,12cyclictest4167302-21taskset22:14:093
393697899192,12cyclictest41553-21taskset22:55:263
3936978991916,2cyclictest0-21swapper/320:10:183
3936978991916,1cyclictest0-21swapper/322:25:183
3936978991915,2cyclictest0-21swapper/321:40:003
3936978991915,2cyclictest0-21swapper/300:35:163
393697899191,2cyclictest0-21swapper/323:26:513
393697899191,15cyclictest115463-21turbostat23:45:013
393697899191,14cyclictest81356-21chrt23:20:513
393697899191,14cyclictest0-21swapper/323:11:003
393697899191,13cyclictest115779-21chrt23:45:093
393697899191,11cyclictest4033697-21taskset20:35:203
3936978991910,4cyclictest685002sleep323:15:153
393697899190,16cyclictest66550irq/131-enp1s0-TxRx-321:58:073
1058012192,3sleep20-21swapper/223:36:472
895012181,3sleep10-21swapper/123:26:001
393697899182,11cyclictest183743-21taskset00:26:023
3936978991815,1cyclictest0-21swapper/319:20:173
3936978991814,2cyclictest0-21swapper/319:55:153
393697899181,14cyclictest0-21swapper/320:04:253
393697899181,13cyclictest0-21swapper/320:30:143
393697899181,11cyclictest17104-21taskset22:40:193
393697899180,9cyclictest152452-21taskset00:05:413
1766812181,3sleep10-21swapper/100:21:521
110722181,3sleep20-21swapper/222:37:262
817672171,4sleep14066340-21diskmemload23:21:351
684622171,3sleep00-21swapper/023:15:120
41596252171,4sleep24066340-21diskmemload22:09:202
41595772171,10sleep14066340-21diskmemload22:09:151
40971332171,7sleep04097144-21ssh21:28:450
40878812171,3sleep00-21swapper/021:21:150
40806562171,3sleep00-21swapper/021:17:300
3936978991714,2cyclictest0-21swapper/321:40:013
3936978991714,1cyclictest0-21swapper/321:00:153
3936978991714,1cyclictest0-21swapper/320:05:173
393697899171,13cyclictest0-21swapper/320:45:003
393697899171,12cyclictest0-21swapper/320:50:143
393697899171,10cyclictest3970946-21taskset19:40:013
393697899171,10cyclictest3964199-21taskset19:30:193
265572171,2sleep00-21swapper/022:46:220
1911312171,10sleep04066340-21diskmemload00:31:210
156022171,10sleep20-21swapper/222:40:202
1533412171,3sleep00-21swapper/000:07:080
1149812171,4sleep14066340-21diskmemload23:44:121
981932161,2sleep00-21swapper/023:32:360
971452161,4sleep14066340-21diskmemload23:30:451
664472160,2sleep00-21swapper/023:12:360
501552161,2sleep00-21swapper/023:01:210
41825692161,2sleep10-21swapper/122:23:511
40190932161,2sleep20-21swapper/220:25:142
3936978991614,1cyclictest0-21swapper/320:55:193
393697899161,12cyclictest4045283-21turbostat20:49:593
393697899160,13cyclictest0-21swapper/320:20:133
393697899160,13cyclictest0-21swapper/319:15:253
279642161,3sleep20-21swapper/222:48:442
22822161,2sleep00-21swapper/022:30:310
1554102161,4sleep14066340-21diskmemload00:10:091
1374462161,2sleep10-21swapper/123:57:111
898912151,2sleep00-21swapper/023:26:430
588932150,3sleep24066340-21diskmemload23:08:012
447562150,3sleep24066340-21diskmemload23:00:042
41444232152,9sleep10-21swapper/121:59:571
41197532151,9sleep04119758-21ssh21:42:330
41100672151,8sleep14066340-21diskmemload21:35:191
41044282151,9sleep04066340-21diskmemload21:32:580
40889742151,3sleep14066340-21diskmemload21:23:051
393697899151,9cyclictest4008249-21taskset20:15:123
3936978991513,1cyclictest0-21swapper/319:45:273
3936978991513,1cyclictest0-21swapper/319:10:163
2005252150,2sleep20-21swapper/200:39:572
1930002156,5sleep2193158-21cat00:35:012
1843992150,2sleep00-21swapper/000:27:280
1460272151,2sleep20-21swapper/200:03:062
1288132150,3sleep20-21swapper/223:51:072
1234862151,8sleep04066340-21diskmemload23:50:040
1042822152,7sleep0104341-21ntpq23:35:210
612002141,9sleep262811-21tr23:10:182
582422141,3sleep14066340-21diskmemload23:06:511
41521332141,2sleep04066340-21diskmemload22:04:500
40988522141,9sleep14100013-21tr21:30:191
40904652141,9sleep14090439-21/usr/sbin/munin21:25:131
40402622141,8sleep24040281-21apt-get20:45:142
39411102141,4sleep23941191-21ntpq19:10:192
23962141,7sleep10-21swapper/122:30:481
1992282141,8sleep04066340-21diskmemload00:37:300
1541672141,8sleep14066340-21diskmemload00:08:291
1216102141,7sleep14066340-21diskmemload23:47:121
1139862141,4sleep2113998-21ssh23:42:252
759242131,5sleep20-21swapper/223:19:592
750302131,8sleep175043-21ssh23:18:221
673592131,3sleep167371-21ssh23:14:131
41840452131,7sleep04066340-21diskmemload22:25:170
41828202130,8sleep04066340-21diskmemload22:24:180
41742702131,6sleep10-21swapper/122:17:491
41507902131,6sleep14066340-21diskmemload22:02:251
40922942130,2sleep20-21swapper/221:25:172
40349732130,3sleep04034992-21apt-get20:40:150
40175342132,5sleep14017574-21ntpq20:20:211
39762842130,2sleep2171rcu_preempt19:45:142
1921782130,1sleep10-21swapper/100:33:191
1703682131,5sleep10-21swapper/100:19:191
1684572131,6sleep00-21swapper/000:16:100
1682102136,3sleep23936971-21cyclictest00:15:442
1221652131,8sleep2122177-21ssh23:48:142
1066842131,6sleep14066340-21diskmemload23:38:181
766512121,5sleep20-21swapper/223:20:162
529272121,3sleep00-21swapper/023:05:140
41908912120,8sleep24190894-21ssh22:29:482
41581682121,5sleep00-21swapper/022:06:490
41341322124,3sleep0151ktimers/021:50:310
41284832121,7sleep04066340-21diskmemload21:49:130
41274202121,7sleep14127428-21sh21:47:181
41198942121,7sleep24119905-21ssh21:42:462
41104182121,5sleep20-21swapper/221:35:242
41055812121,7sleep24105602-21ls21:35:002
40562572121,7sleep04056075-21/usr/sbin/munin21:00:150
40400002121,7sleep14040015-21latency_hist20:45:011
290312121,5sleep00-21swapper/022:50:070
1837842121,3sleep20-21swapper/200:26:142
1773682121,5sleep00-21swapper/000:23:110
1608682120,5sleep014-21ksoftirqd/000:11:370
1380452121,6sleep00-21swapper/023:58:100
1241822121,3sleep1125030-21kthreadcore23:50:171
100282121,4sleep14066340-21diskmemload22:35:361
893342111,3sleep20-21swapper/223:25:412
770252110,7sleep078637-21kthreadcore23:20:170
519022111,4sleep14066340-21diskmemload23:04:271
433742111,4sleep24066340-21diskmemload22:58:102
41757832111,3sleep20-21swapper/222:20:022
41665522111,4sleep14066340-21diskmemload22:12:521
41438742111,3sleep20-21swapper/221:59:052
41358172111,4sleep24066340-21diskmemload21:53:282
41334932110,2sleep10-21swapper/121:50:241
41282442110,4sleep20-21swapper/221:48:432
410992111,5sleep141155-21ntpq22:55:211
41099192111,5sleep04110065-21ntpq21:35:180
40880882111,3sleep20-21swapper/221:21:372
40796622111,3sleep10-21swapper/121:15:511
40730492111,7sleep04073060-21sh21:12:530
40720562111,4sleep14066340-21diskmemload21:11:191
40296152112,4sleep24029634-21apt-get20:35:122
39656272111,5sleep23965646-21apt-get19:35:122
374872111,5sleep037757-21cut22:55:160
359952111,3sleep20-21swapper/222:54:032
346202111,4sleep14066340-21diskmemload22:51:411
275582111,3sleep14066340-21diskmemload22:48:031
190202111,4sleep04066340-21diskmemload22:42:210
180642111,4sleep14066340-21diskmemload22:40:391
1516292111,5sleep2151678-21ntpq00:05:212
1454122111,4sleep14066340-21diskmemload00:02:041
1151302111,4sleep04066340-21diskmemload23:44:270
982162101,3sleep20-21swapper/223:32:392
41735152101,3sleep04066340-21diskmemload22:16:270
41603412100,6sleep04066340-21diskmemload22:10:070
41410412101,4sleep04066340-21diskmemload21:55:190
40673332101,5sleep24068294-21ssh21:10:192
40188162101,3sleep00-21swapper/020:25:120
39815342101,4sleep03981553-21apt-get19:50:140
39801152101,4sleep03980256-21ntpq19:45:200
39760022101,5sleep13976174-21cat19:45:011
39703992100,3sleep172-21kcompactd019:36:271
39624902101,4sleep23964198-21sed19:30:192
39590942101,5sleep23959151-21ntpq19:25:222
39484792102,3sleep13948520-21ntpq19:15:191
25772101,2sleep20-21swapper/222:31:042
1985202101,3sleep14066340-21diskmemload00:36:111
1841772101,3sleep10-21swapper/100:27:051
1762102101,6sleep24066340-21diskmemload00:20:582
1553742101,3sleep24066340-21diskmemload00:10:072
1461842101,3sleep00-21swapper/000:03:190
1390392101,2sleep20-21swapper/223:59:482
1220932101,3sleep04066340-21diskmemload23:48:080
101962100,2sleep00-21swapper/022:35:560
4149959291,2sleep20-21swapper/222:00:582
4074591291,4sleep24074619-21apt-get21:15:122
4055099291,4sleep14050781-21/usr/sbin/munin20:55:271
4050517291,4sleep21-21systemd20:55:012
3987113291,4sleep13987271-21head19:55:151
3972608291,4sleep03974402-21grep19:40:200
3964340291,4sleep03964378-21ntpq19:30:210
3960913291,4sleep13962489-21grep19:30:181
3959040291,4sleep13959087-21ntpq19:25:211
3949786291,2sleep10-21swapper/119:20:121
3945205291,4sleep23946890-21cut19:15:162
3938704291,4sleep03940485-21tr19:10:180
4166182280,2sleep24066340-21diskmemload22:12:122
4065360281,3sleep24065409-21ntpq21:05:222
4049795281,3sleep11-21systemd20:50:261
4045084280,1sleep00-21swapper/020:47:400
4044115281,3sleep14040263-21/usr/sbin/munin20:45:211
4028811280,1sleep10-21swapper/120:31:421
4014279281,3sleep24015945-21kthreadcore20:20:182
4007299281,3sleep04002981-21/usr/sbin/munin20:10:290
4001718281,3sleep14001766-21ntpq20:05:211
3997292280,2sleep10-21swapper/120:03:211
3981815280,4sleep20-21swapper/219:50:152
3971420280,1sleep20-21swapper/219:40:172
3951056281,3sleep23952804-21kthreadcore19:20:172
4118050270,2sleep10-21swapper/121:40:251
4061335270,4sleep13942302-21kworker/1:0+mm_percpu_wq21:05:051
4056074270,2sleep20-21swapper/221:00:142
4055665270,1sleep00-21swapper/020:58:260
4050387270,1sleep00-21swapper/020:53:210
4034634272,3sleep0151ktimers/020:39:230
4007868270,1sleep10-21swapper/120:13:311
3997684270,2sleep00-21swapper/020:05:120
3994185271,3sleep00-21swapper/020:00:200
3990719271,2sleep23990808-21ntpq19:55:202
3976262270,2sleep10-21swapper/119:45:131
4066117260,1sleep00-21swapper/021:07:200
4061005260,1sleep10-21swapper/121:04:001
4055457260,1sleep20-21swapper/220:55:482
4039941260,1sleep20-21swapper/220:44:192
4024090261,2sleep20-21swapper/220:30:162
4018557260,3sleep10-21swapper/120:25:011
4017680261,2sleep00-21swapper/020:20:230
4012568260,3sleep14012734-21wc20:15:271
4009185260,2sleep00-21swapper/020:15:180
4002401260,1sleep20-21swapper/220:06:022
3991480260,1sleep00-21swapper/019:55:460
3982984261,2sleep10-21swapper/119:50:181
3965620260,4sleep00-21swapper/019:35:060
3958951260,2sleep00-21swapper/019:25:200
3953644261,2sleep00-21swapper/019:20:200
3942704260,2sleep00-21swapper/019:15:020
4034098250,1sleep10-21swapper/120:35:271
4006874250,1sleep20-21swapper/220:10:232
3991910250,1sleep20-21swapper/220:00:012
4023803240,1sleep00-21swapper/020:30:010
4008598240,1sleep20-21swapper/220:15:162
3937429240,1sleep10-21swapper/119:10:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional