You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 02:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Fri Nov 14, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38659019912676,1252cyclictest139302sleep311:05:023
38659019912390,1235cyclictest40528532sleep309:30:013
38659019912100,1207cyclictest39811162sleep308:39:573
38659019910750,1071cyclictest41588352sleep310:34:563
3865901993027,1cyclictest0-21swapper/309:40:003
386590199198,8cyclictest710612sleep311:40:133
386590199198,8cyclictest3962318-21turbostat08:25:013
39452712181,14sleep03946873-21sort08:10:180
386590199171,12cyclictest3872115-21turbostat07:15:003
41359872162,7sleep04137116-21kthreadcore10:20:200
386590199166,7cyclictest38202-21turbostat11:20:003
386590199165,7cyclictest685012sleep311:35:243
3865901991613,1cyclictest0-21swapper/310:40:183
386590199161,12cyclictest160234-21taskset12:35:013
39114272150,6sleep13911626-21anacron07:40:221
3865901991513,1cyclictest0-21swapper/309:07:093
3865901991513,1cyclictest0-21swapper/308:27:323
3865901991513,0cyclictest0-21swapper/311:25:173
3865901991512,1cyclictest0-21swapper/309:15:183
386590199151,11cyclictest103033-21turbostat12:00:003
386590199145,6cyclictest31042sleep310:55:253
386590199143,8cyclictest4025713-21taskset09:10:223
386590199141,9cyclictest791442sleep311:45:143
386590199141,9cyclictest39688512sleep308:30:123
3865901991412,1cyclictest0-21swapper/312:08:083
3865901991412,1cyclictest0-21swapper/311:00:123
3865901991412,0cyclictest0-21swapper/311:14:203
3865901991411,1cyclictest0-21swapper/307:50:173
386590199141,10cyclictest4149164-21taskset10:26:153
386590199141,10cyclictest4092892-21taskset09:53:093
386590199141,10cyclictest4044780-21taskset09:25:123
386590199141,10cyclictest3937133-21chrt08:03:033
386590199141,10cyclictest38448-21chrt11:20:003
386590199141,10cyclictest152219-21taskset12:30:093
386590199141,10cyclictest142623-21chrt12:22:033
386590199141,10cyclictest142623-21chrt12:22:033
386590199141,10cyclictest122203-21taskset12:10:173
41021632131,8sleep14102197-21apt-get10:00:131
386590199134,7cyclictest4183622-21taskset10:50:033
386590199134,6cyclictest1510672sleep312:28:053
386590199133,7cyclictest4083443-21taskset09:45:243
386590199131,9cyclictest4175577-21taskset10:45:023
386590199131,9cyclictest4140801-21chrt10:20:303
386590199131,9cyclictest4133645-21chrt10:17:453
386590199131,9cyclictest4108183-21taskset10:00:433
386590199131,9cyclictest4000462-21taskset08:55:153
386590199131,9cyclictest3999565-21taskset08:50:433
386590199131,9cyclictest3992883-21chrt08:45:223
386590199131,9cyclictest3981842-21taskset08:40:173
386590199131,9cyclictest3918309-21chrt07:48:203
386590199131,9cyclictest3893481-21taskset07:30:143
386590199131,9cyclictest3879865-21chrt07:15:233
386590199131,8cyclictest39438132sleep308:10:133
3865901991312,0cyclictest0-21swapper/312:17:403
3865901991312,0cyclictest0-21swapper/312:17:403
3865901991311,1cyclictest0-21swapper/307:58:573
386590199131,10cyclictest93422-21taskset11:51:183
3865901991311,0cyclictest0-21swapper/307:43:333
3865901991311,0cyclictest0-21swapper/307:36:373
3865901991310,1cyclictest0-21swapper/309:55:153
3865901991310,1cyclictest0-21swapper/309:55:153
3865901991310,1cyclictest0-21swapper/309:41:073
386590199130,9cyclictest41591442sleep310:35:063
386590199130,10cyclictest62380-21taskset11:34:193
386590199130,10cyclictest4125489-21chrt10:12:413
386590199130,10cyclictest3943330-21taskset08:07:073
386590199130,10cyclictest3892879-21taskset07:27:133
190898130,10rtkit-daemon3872259-21munin-node10:13:240
386590199124,6cyclictest3955839-21taskset08:17:493
386590199121,9cyclictest4006956-21taskset09:00:123
386590199121,9cyclictest3885892-21taskset07:20:203
386590199121,8cyclictest4042141-21taskset09:20:233
386590199120,10cyclictest4116442-21taskset10:05:513
386590199120,10cyclictest109791-21chrt12:01:563
38598122120,6sleep30-21swapper/307:05:143
190898120,8rtkit-daemon109352-21ssh12:00:590
1585182120,4sleep1158526-21ssh12:31:361
575632110,4sleep259699-21cut11:30:222
439032110,9sleep044041-21cat11:20:220
41036582111,3sleep20-21swapper/210:00:182
190898110,6rtkit-daemon438-21wpa_supplicant07:15:121
190898110,6rtkit-daemon3917581-21seq07:45:211
41669182100,3sleep24166924-21ssh10:39:532
41251022101,6sleep24125110-21ssh10:11:522
40922362100,7sleep14092244-21ssh09:51:401
40606342101,6sleep14060642-21ssh09:34:371
39777622100,7sleep23979963-21cut08:35:202
39247312101,6sleep13924744-21munin-plugin-st07:55:011
38744812100,7sleep03874538-21sort07:15:130
385512102,4sleep238573-21apt-get11:20:122
190898100,6rtkit-daemon4118081-21ssh10:09:400
190898100,6rtkit-daemon4084-21ssh10:56:290
190898100,5rtkit-daemon3925019-21awk07:55:121
190898100,3rtkit-daemon667-21snmpd10:23:552
190898100,3rtkit-daemon4190217-21ssh10:52:090
190898100,3rtkit-daemon3826652-21munin-node07:14:181
1004382101,7sleep195304-21/usr/sbin/munin11:55:221
93936290,1sleep04019797-21diskmemload11:52:320
68839291,6sleep10-21swapper/111:35:421
4142673296,2sleep1331ktimers/110:24:581
4118063290,6sleep14118071-21ssh10:09:391
4101117291,6sleep14101123-21ssh09:58:251
4101117291,6sleep14101123-21ssh09:58:251
4092148291,5sleep24092156-21ssh09:51:292
4085451291,5sleep20-21swapper/209:50:012
4046659290,7sleep00-21swapper/009:25:180
4028341292,3sleep04028378-21apt-get09:15:130
3956323290,6sleep13956333-21cpu08:20:131
19089890,5rtkit-daemon667-21snmpd12:36:342
19089890,4rtkit-daemon86398-21ssh11:48:450
19089890,4rtkit-daemon61834-21ssh11:33:041
19089890,4rtkit-daemon4132997-21ssh10:16:160
19089890,4rtkit-daemon4077582-21apt-get09:45:120
19089890,4rtkit-daemon4051461-21ssh09:27:211
19089890,4rtkit-daemon4019797-21diskmemload11:23:451
19089890,4rtkit-daemon4014371-21irqcore09:05:152
19089890,4rtkit-daemon3994079-21df_inode08:50:142
19089890,4rtkit-daemon100687-21python311:55:240
19089890,1rtkit-daemon0-21swapper/208:26:252
127728290,6sleep0127771-21ssh12:15:120
127728290,6sleep0127771-21ssh12:15:120
93413280,1sleep20-21swapper/211:51:182
85939280,1sleep20-21swapper/211:47:412
77745281,5sleep277751-21ssh11:42:322
64288281,5sleep065424-21kthreadcore11:35:170
46953281,4sleep047106-21df_inode11:25:150
4174043280,6sleep04174050-21sh10:42:400
4158482281,4sleep14158490-21ssh10:34:031
4100098280,5sleep04100105-21sh09:56:000
4100098280,5sleep04100105-21sh09:56:000
4020321281,4sleep24020399-21fschecks_time09:10:142
4005305281,4sleep10-21swapper/108:55:221
3993890281,3sleep03993923-21apt-get08:50:120
3988118280,6sleep13989109-21/usr/sbin/munin08:45:171
3982687280,5sleep03983880-21cut08:40:190
3975147280,4sleep13975293-21df08:35:151
3962565281,3sleep03962598-21apt-get08:25:160
3950044280,4sleep23950077-21apt-get08:15:132
3906501280,5sleep23907272-21irqrtprio07:40:152
3904832280,4sleep23904964-21file07:35:222
34974280,6sleep135580-21/usr/sbin/munin11:15:191
19089880,5rtkit-daemon1928-21sendmail-mta08:49:552
19089880,4rtkit-daemon8656-21grep11:00:220
19089880,4rtkit-daemon81364-21packagekitd10:48:290
19089880,4rtkit-daemon674-21lldpd11:18:282
19089880,4rtkit-daemon674-21lldpd09:43:572
19089880,4rtkit-daemon671-21ntpd08:18:541
19089880,4rtkit-daemon671-21ntpd07:05:171
19089880,4rtkit-daemon667-21snmpd12:28:460
19089880,4rtkit-daemon667-21snmpd10:00:510
19089880,4rtkit-daemon667-21snmpd09:51:510
19089880,4rtkit-daemon667-21snmpd09:20:230
19089880,4rtkit-daemon667-21snmpd07:25:501
19089880,4rtkit-daemon436-21systemd-logind07:36:381
19089880,4rtkit-daemon4156477-21systemctl10:30:210
19089880,4rtkit-daemon4076962-21ssh09:44:200
19089880,4rtkit-daemon4006128-21sh08:56:472
19089880,4rtkit-daemon3937725-21df08:05:141
19089880,4rtkit-daemon3872259-21munin-node12:06:330
19089880,4rtkit-daemon3872259-21munin-node11:42:460
19089880,4rtkit-daemon3872259-21munin-node11:40:021
19089880,4rtkit-daemon3872259-21munin-node11:12:322
19089880,4rtkit-daemon1928-21sendmail-mta08:01:001
19089880,4rtkit-daemon125710-21ssh12:10:580
19089880,1rtkit-daemon0-21swapper/111:27:491
160313281,4sleep1160346-21awk12:35:121
159525280,1sleep20-21swapper/212:33:572
136558281,4sleep0137184-21sed12:20:170
136558281,4sleep0137184-21sed12:20:170
120187280,5sleep1120990-21irqrtprio12:10:161
118916283,3sleep1118921-21sh12:08:521
87304270,4sleep187456-21df11:50:131
6446271,4sleep10-21swapper/111:00:201
52741271,4sleep252749-21ssh11:25:452
4183705270,3sleep14183740-21apt-get10:50:141
4165618270,1sleep10-21swapper/110:36:561
4164658271,3sleep04164809-21cat10:35:270
4158861271,3sleep24158888-21apt10:35:002
4149761270,1sleep10-21swapper/110:27:381
4095010270,5sleep24096194-21cut09:55:182
4095010270,5sleep24096194-21cut09:55:182
4070865270,5sleep14072788-21cut09:40:181
4050951271,3sleep24050958-21diskmemload09:26:082
4035902270,1sleep20-21swapper/209:19:232
4025839270,4sleep04026049-21users09:10:240
4014003270,4sleep04014143-21cut09:05:150
4000157271,3sleep04000190-21apt-get08:55:130
3993549270,1sleep00-21swapper/008:49:000
3974164270,4sleep23974369-21grep08:30:232
3961718270,4sleep03961935-21users08:20:240
3930310271,4sleep03930480-21cat07:55:240
3906270270,4sleep03906314-21tune2fs07:40:130
3860195270,4sleep23860387-21cut07:05:172
19089870,4rtkit-daemon81364-21packagekitd11:11:091
19089870,4rtkit-daemon671-21ntpd08:25:111
19089870,4rtkit-daemon667-21snmpd07:20:341
19089870,4rtkit-daemon1928-21sendmail-mta09:00:402
19089870,3rtkit-daemon81364-21packagekitd10:25:492
19089870,3rtkit-daemon674-21lldpd09:34:272
19089870,3rtkit-daemon667-21snmpd07:30:191
19089870,3rtkit-daemon4134602-21ssh10:19:551
19089870,3rtkit-daemon3872259-21munin-node11:05:020
19089870,3rtkit-daemon3872259-21munin-node08:41:022
19089870,3rtkit-daemon3872259-21munin-node08:10:121
19089870,3rtkit-daemon26421-21cut11:10:200
19089870,3rtkit-daemon1928-21sendmail-mta09:35:512
19089870,3rtkit-daemon152436-21df_inode12:30:130
143672270,5sleep1143679-21diskmemload12:24:351
143672270,5sleep1143679-21diskmemload12:24:351
111643271,3sleep2111680-21apt-get12:05:122
86101260,4sleep186109-21ssh11:48:081
5591260,1sleep20-21swapper/210:59:572
54981260,3sleep055018-21apt-get11:30:160
4183295262,3sleep24006914-21kworker/u16:1+xprtiod10:49:472
4177141260,3sleep14177179-21kthreadcore10:45:171
4143338260,3sleep04143521-21if_eno110:25:170
4128903260,3sleep24130978-21cut10:15:172
4117484260,1sleep20-21swapper/210:08:162
4085416260,1sleep10-21swapper/109:49:561
4029826261,2sleep10-21swapper/109:15:181
4026847260,1sleep10-21swapper/109:12:171
3981387260,1sleep10-21swapper/108:40:041
3975111263,1sleep00-21swapper/008:35:140
3955935260,1sleep00-21swapper/008:19:030
3937738261,2sleep20-21swapper/208:05:152
3918738260,3sleep03918890-21df_inode07:50:150
3904705260,3sleep03904782-21ntp_states07:35:200
3887250260,2sleep00-21swapper/007:25:150
3880800260,3sleep03880952-21df_inode07:20:140
3867511260,4sleep03868806-21kthreadcore07:10:160
3859838260,2sleep00-21swapper/007:05:150
37602260,1sleep00-21swapper/011:18:420
150630260,4sleep1150636-21ssh12:27:031
135977260,3sleep2135517-21apt-get12:20:072
135977260,3sleep2135517-21apt-get12:20:072
130356261,2sleep20-21swapper/212:15:182
130356261,2sleep20-21swapper/212:15:182
12105260,4sleep20-21swapper/211:01:172
111101261,3sleep2111163-21apt-key12:05:002
110692260,3sleep1110702-21ssh12:04:071
101828260,1sleep20-21swapper/211:57:152
70245251,2sleep2431ktimers/211:38:562
4188972250,2sleep20-21swapper/210:50:232
4173528250,3sleep14142326-21kworker/u16:3+rpciod10:41:251
4167728251,2sleep20-21swapper/210:40:172
4067756250,3sleep14006914-21kworker/u16:1+rpciod09:37:091
4067497250,3sleep00-21swapper/009:36:280
4042997251,2sleep20-21swapper/209:21:482
4038345251,2sleep10-21swapper/109:20:181
4018890250,3sleep10-21swapper/109:05:201
3999558250,1sleep10-21swapper/108:50:381
3974621250,1sleep00-21swapper/008:32:060
3974037250,2sleep10-21swapper/108:30:221
3949787250,3sleep2423-21dbus-daemon08:15:012
3930667250,1sleep20-21swapper/207:55:462
3914994251,2sleep00-21swapper/007:45:180
3912819250,2sleep20-21swapper/207:45:162
3898991250,1sleep00-21swapper/007:30:280
3894346251,2sleep20-21swapper/207:30:182
3892866250,1sleep20-21swapper/207:27:052
3886157251,2sleep2421rcuc/207:20:242
149521251,2sleep20-21swapper/212:25:242
14012250,2sleep10-21swapper/111:05:131
133533251,2sleep10-21swapper/112:15:291
133533251,2sleep10-21swapper/112:15:291
120991250,3sleep2122202-21tr12:10:172
4192012240,1sleep10-21swapper/110:55:191
4118467240,2sleep14118469-21ssh10:10:031
4052863240,3sleep00-21swapper/009:30:030
4007120240,2sleep10-21swapper/109:00:151
4006963240,1sleep00-21swapper/009:00:140
3958003240,2sleep20-21swapper/208:20:182
3937547240,2sleep00-21swapper/008:05:130
3937195240,3sleep20-21swapper/208:03:512
3937080240,3sleep00-21swapper/008:02:210
3923663240,1sleep20-21swapper/207:50:212
3880153240,3sleep20-21swapper/207:15:512
3870999240,2sleep23871171-21strings07:10:182
167041240,3sleep00-21swapper/012:37:320
13952240,2sleep20-21swapper/211:05:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional