You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 05:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Fri Nov 28, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2517069912910,1288cyclictest7050452sleep300:14:563
2517069912660,1262cyclictest6406102sleep323:34:573
2517069912610,1258cyclictest3926072sleep320:59:563
6780622230,1sleep10-21swapper/123:55:181
25170699181,13cyclictest673239-21taskset23:55:123
190898180,7rtkit-daemon298094-21apt-get19:45:111
25170699176,8cyclictest304101-21turbostat19:50:003
251706991714,1cyclictest0-21swapper/322:00:153
190898170,3rtkit-daemon437222-21ssh21:28:530
251706991614,1cyclictest0-21swapper/323:00:003
251706991613,1cyclictest0-21swapper/323:28:003
251706991613,1cyclictest0-21swapper/323:20:173
251706991613,1cyclictest0-21swapper/321:10:243
251706991613,1cyclictest0-21swapper/320:10:003
251706991613,1cyclictest0-21swapper/320:00:153
25170699161,12cyclictest616229-21turbostat23:19:593
25170699161,11cyclictest648677-21turbostat23:40:013
25170699161,11cyclictest478562-21taskset21:55:143
5128842151,7sleep2514935-21kthreadcore22:15:182
25170699158,4cyclictest6003362sleep323:10:133
25170699157,6cyclictest354876-21turbostat20:30:003
25170699156,7cyclictest421402-21turbostat21:24:563
25170699155,7cyclictest541ktimers/319:40:003
251706991512,1cyclictest0-21swapper/319:15:243
251706991512,1cyclictest0-21swapper/300:04:493
25170699151,11cyclictest559611-21taskset22:45:153
25170699151,11cyclictest543228-21taskset22:35:143
25170699151,11cyclictest444327-21taskset21:31:423
25170699151,10cyclictest511049-21taskset22:15:143
7454072141,4sleep2745423-21latency_hist00:39:592
25170699149,3cyclictest4534212sleep321:39:113
25170699145,7cyclictest688956-21turbostat00:09:553
25170699145,6cyclictest4773312sleep321:53:273
25170699141,9cyclictest524323-21taskset22:20:233
251706991413,0cyclictest0-21swapper/322:30:083
251706991413,0cyclictest0-21swapper/322:30:083
251706991412,1cyclictest0-21swapper/323:41:333
251706991412,1cyclictest0-21swapper/320:49:333
251706991412,1cyclictest0-21swapper/319:32:033
25170699141,11cyclictest251856-21taskset19:10:023
251706991411,1cyclictest0-21swapper/322:05:203
251706991411,1cyclictest0-21swapper/319:58:393
25170699141,10cyclictest737271-21taskset00:34:433
25170699141,10cyclictest721622-21taskset00:25:143
25170699141,10cyclictest657095-21taskset23:45:103
25170699141,10cyclictest591052-21taskset23:03:393
25170699141,10cyclictest557140-21chrt22:40:433
25170699141,10cyclictest532614-21taskset22:25:243
25170699141,10cyclictest461804-21taskset21:44:533
25170699141,10cyclictest414130-21taskset21:15:153
25170699141,10cyclictest404681-21taskset21:05:253
25170699141,10cyclictest380376-21taskset20:50:153
25170699141,10cyclictest367138-21chrt20:35:363
25170699141,10cyclictest360142-21chrt20:30:193
25170699140,11cyclictest738604-21taskset00:35:153
25170699140,11cyclictest599700-21taskset23:09:563
25170699140,11cyclictest367748-21taskset20:40:013
190898140,4rtkit-daemon541381-21ssh22:31:380
190898140,4rtkit-daemon541381-21ssh22:31:380
4448012130,10sleep1444809-21ssh21:32:531
3440172131,9sleep0345150-21grep20:20:200
3440172131,9sleep0345150-21grep20:20:200
25170699131,9cyclictest572694-21taskset22:50:203
25170699131,9cyclictest271822-21taskset19:20:243
251706991312,0cyclictest0-21swapper/321:45:483
251706991312,0cyclictest0-21swapper/300:15:453
251706991311,1cyclictest0-21swapper/321:27:463
251706991311,1cyclictest0-21swapper/319:28:303
25170699131,10cyclictest720127-21taskset00:22:213
25170699131,10cyclictest672467-21taskset23:54:003
25170699131,10cyclictest297566-21taskset19:41:553
25170699130,10cyclictest3929052sleep321:00:183
25170699130,10cyclictest342545-21taskset20:20:023
25170699130,10cyclictest342545-21taskset20:20:023
190898130,9rtkit-daemon735442-21ssh00:30:350
4520582121,9sleep1452062-21ssh21:36:021
25170699122,7cyclictest3296842sleep320:10:133
25170699121,9cyclictest310159-21taskset19:52:453
25170699120,9cyclictest335968-21chrt20:15:013
25170699120,10cyclictest502325-21taskset22:10:013
190898120,7rtkit-daemon454372-21grep21:40:140
190898120,4rtkit-daemon477864-21ssh21:54:450
6794112111,7sleep2679419-21ssh23:56:322
6243062111,7sleep0624415-21apt-key23:25:010
5911002110,8sleep1591106-21ssh23:03:461
5169882111,7sleep0516996-21ssh22:16:200
4867702110,9sleep2487057-21sed22:00:152
3985622110,1sleep0310290-21kworker/0:1+mm_percpu_wq21:00:350
2515612110,6sleep1251590-21munin-run19:10:011
190898110,4rtkit-daemon81364-21packagekitd19:52:011
7364802101,6sleep2736486-21ssh00:32:552
6795832100,7sleep0679591-21ssh23:56:550
6425872100,3sleep2644535-21kthreadcore23:35:172
6218182100,1sleep2616557-21/usr/sbin/munin23:20:222
5563992100,7sleep0556477-21/usr/sbin/munin22:40:210
4782852100,3sleep0478289-21ssh21:55:080
2455892101,5sleep30-21swapper/319:05:143
190898100,7rtkit-daemon667-21snmpd21:12:101
190898100,6rtkit-daemon468271-21ssh21:46:050
190898100,6rtkit-daemon340380-21cut20:15:181
190898100,6rtkit-daemon1-21systemd20:30:371
190898100,5rtkit-daemon322114-21memory20:00:191
190898100,4rtkit-daemon566764-21ssh22:49:100
190898100,4rtkit-daemon545579-21grep22:35:180
190898100,4rtkit-daemon258197-21munin-node22:16:431
190898100,3rtkit-daemon615107-21ssh23:17:260
190898100,3rtkit-daemon421-21avahi-daemon23:08:560
705071291,5sleep2705082-21cron00:15:002
526274290,1sleep10-21swapper/122:24:111
460933291,6sleep2460939-21ssh21:42:522
365476291,6sleep1361438-21/usr/sbin/munin20:35:191
297252291,5sleep2297437-21users19:40:232
260773290,6sleep0261739-21cut19:15:170
245615291,4sleep0245771-21diskstats19:05:150
19089890,5rtkit-daemon438-21wpa_supplicant21:00:291
19089890,5rtkit-daemon1928-21sendmail-mta19:13:081
19089890,4rtkit-daemon706925-21ssh00:15:190
19089890,4rtkit-daemon647063-21ssh23:36:130
19089890,4rtkit-daemon420203-21ssh21:17:141
19089890,2rtkit-daemon688137-21ssh00:03:080
19089890,1rtkit-daemon0-21swapper/021:34:520
728975281,4sleep1728983-21ssh00:29:171
720583280,6sleep1720590-21ssh00:23:281
657112280,5sleep1657149-21apt-get23:45:131
624900281,3sleep0624933-21apt-get23:25:130
575217280,5sleep1299-21systemd-journal22:55:001
567093282,3sleep244-21ksoftirqd/222:50:002
558253281,4sleep1558261-21ssh22:43:181
539669280,6sleep10-21swapper/122:30:201
539669280,6sleep10-21swapper/122:30:191
534701281,4sleep0534707-21ssh22:29:550
532475280,5sleep2532613-21timerandwakeup22:25:242
502675281,4sleep0502872-21/usr/sbin/munin22:10:150
484525281,4sleep2484532-21sh21:56:272
479341280,2sleep10-21swapper/121:55:171
468749280,5sleep1468757-21ssh21:47:161
437551280,1sleep10-21swapper/121:29:371
360284281,4sleep0360338-21ntp_states20:30:210
336470280,1sleep20-21swapper/220:15:172
278251281,4sleep1278465-21grep19:25:231
278125281,3sleep0278250-21python319:25:220
19089880,5rtkit-daemon667-21snmpd23:44:180
19089880,5rtkit-daemon1928-21sendmail-mta21:05:081
19089880,4rtkit-daemon737597-21ssh00:35:031
19089880,4rtkit-daemon727576-21ssh00:25:530
19089880,4rtkit-daemon703893-21ssh00:12:090
19089880,4rtkit-daemon674-21lldpd22:50:200
19089880,4rtkit-daemon671295-21ssh23:51:190
19089880,4rtkit-daemon667-21snmpd23:10:410
19089880,4rtkit-daemon667-21snmpd19:36:311
19089880,4rtkit-daemon667-21snmpd19:20:211
19089880,4rtkit-daemon583881-21ssh23:00:080
19089880,4rtkit-daemon448677-21tr21:35:190
19089880,4rtkit-daemon436-21systemd-logind20:53:371
19089880,4rtkit-daemon429114-21ssh21:24:230
19089880,4rtkit-daemon332659-21cut20:10:181
19089880,4rtkit-daemon261659-21irqrtprio19:15:171
19089880,4rtkit-daemon258197-21munin-node23:33:090
19089880,4rtkit-daemon258197-21munin-node19:55:021
19089880,1rtkit-daemon0-21swapper/122:12:451
721580270,3sleep2721617-21apt-get00:25:142
712738270,2sleep2712746-21ssh00:19:052
671996270,2sleep2672002-21ssh23:53:002
671063270,5sleep1671071-21ssh23:50:421
664144270,5sleep2664151-21sh23:48:272
573520270,4sleep2573528-21ssh22:51:082
541160270,5sleep20-21swapper/222:31:052
541160270,5sleep20-21swapper/222:31:052
526043270,5sleep2526051-21ssh22:23:322
477902270,5sleep20-21swapper/221:54:502
476069270,5sleep1476073-21ssh21:50:291
413676271,3sleep0413840-21diskstats21:15:130
411026270,4sleep0411305-21grep21:10:240
392611271,3sleep2392625-21munin-plugin-st21:00:002
342409270,3sleep2342034-21apt-get20:20:022
342409270,3sleep2342034-21apt-get20:20:022
311202271,3sleep0311333-21tail19:55:150
304342270,3sleep0304375-21apt-get19:50:150
266647271,3sleep2266680-21apt-get19:20:162
258928270,4sleep2260138-21hddtemp_smartct19:15:022
245972270,4sleep2246207-21sed19:05:172
19089870,4rtkit-daemon81364-21packagekitd22:26:211
19089870,4rtkit-daemon671-21ntpd20:40:061
19089870,4rtkit-daemon671-21ntpd20:26:131
19089870,4rtkit-daemon671-21ntpd00:07:310
19089870,4rtkit-daemon667-21snmpd19:31:071
19089870,4rtkit-daemon473-21gmain20:21:271
19089870,4rtkit-daemon473-21gmain20:21:271
19089870,4rtkit-daemon473-21gmain00:20:550
19089870,3rtkit-daemon667-21snmpd23:45:470
19089870,3rtkit-daemon575558-21apt-get22:55:120
19089870,3rtkit-daemon501992-21ssh22:09:431
19089870,3rtkit-daemon258197-21munin-node22:07:122
19089870,3rtkit-daemon258197-21munin-node20:56:441
19089870,1rtkit-daemon0-21swapper/120:06:431
744728260,3sleep0744736-21ssh00:38:260
737072260,1sleep10-21swapper/100:34:181
711333261,2sleep1667-21snmpd00:15:471
704347260,1sleep10-21swapper/100:13:171
696099260,1sleep20-21swapper/200:07:542
689465260,1sleep10-21swapper/100:05:131
683109260,4sleep2685087-21kthreadcore00:00:212
655689260,1sleep10-21swapper/123:42:251
654580260,3sleep2654828-21users23:40:262
641779260,3sleep1642584-21kthreadcore23:35:161
639762260,1sleep20-21swapper/223:33:002
625082260,3sleep2625111-21grep23:25:152
615580260,1sleep10-21swapper/123:18:331
607666260,1sleep20-21swapper/223:14:052
600533260,3sleep1600694-21ssh23:10:151
589149260,4sleep2589211-21open_files23:00:222
582957260,4sleep20-21swapper/222:59:102
565394260,4sleep1565402-21ssh22:46:031
557911260,1sleep2557919-21ssh22:42:352
543590260,4sleep10-21swapper/122:35:171
487845261,2sleep00-21swapper/022:00:170
444963260,4sleep2444970-21sshd21:33:142
428626260,1sleep20-21swapper/221:23:132
420359260,3sleep2420367-21ssh21:17:362
412481260,1sleep20-21swapper/221:13:072
399202260,3sleep0399349-21df21:05:150
367240260,1sleep00-21swapper/020:36:550
323539261,3sleep2323610-21forks20:05:152
293606262,2sleep00-21swapper/019:40:170
279598260,1sleep00-21swapper/019:30:150
19089860,3rtkit-daemon473-21gmain20:45:151
19089860,2rtkit-daemon0-21swapper/119:40:321
720432250,3sleep20-21swapper/200:23:092
687052250,3sleep10-21swapper/100:00:321
640343251,2sleep10-21swapper/123:34:231
632538250,1sleep10-21swapper/123:29:551
617036250,3sleep1617794-21cut23:20:171
609412251,1sleep20-21swapper/223:15:172
575930251,2sleep10-21swapper/122:55:161
543213251,1sleep20-21swapper/222:35:142
525895250,2sleep0525903-21ssh22:23:130
502444250,3sleep2502598-21ssh22:10:132
501104250,3sleep00-21swapper/022:07:420
436336250,3sleep20-21swapper/221:26:452
427650251,2sleep10-21swapper/121:20:511
404496250,2sleep2404680-21cat21:05:252
386123250,1sleep00-21swapper/020:52:010
386105250,1sleep20-21swapper/220:51:472
380082250,3sleep0380217-21cat20:50:010
361029250,3sleep20-21swapper/220:33:032
353977250,3sleep20-21swapper/220:25:232
341199250,1sleep00-21swapper/020:15:230
335710250,1sleep20-21swapper/220:14:492
329720250,3sleep0329876-21diskstats20:10:140
322857250,1sleep00-21swapper/020:01:170
312321250,3sleep20-21swapper/219:55:172
298052250,1sleep20-21swapper/219:45:042
281101251,2sleep20-21swapper/219:30:172
269665251,2sleep0271579-21latency19:20:220
257904250,1sleep00-21swapper/019:14:330
252990251,2sleep20-21swapper/219:10:182
593889240,2sleep10-21swapper/123:05:181
592188240,2sleep2591973-21/usr/sbin/munin23:05:152
492791240,2sleep10-21swapper/122:01:591
461864240,2sleep2462112-21ssh21:45:012
459492240,2sleep10-21swapper/121:40:231
453158240,2sleep2453164-21ssh21:38:392
386649240,1sleep00-21swapper/020:55:140
375492240,2sleep20-21swapper/220:45:182
373597240,3sleep00-21swapper/020:42:400
361437240,1sleep20-21swapper/220:35:122
323407240,1sleep00-21swapper/020:05:130
303840240,3sleep00-21swapper/019:46:540
286468240,2sleep2286970-21kthreadcore19:35:182
272919240,1sleep20-21swapper/219:25:122
392893230,1sleep20-21swapper/221:00:172
367484230,1sleep20-21swapper/220:40:012
349142230,1sleep00-21swapper/020:25:170
322353230,1sleep20-21swapper/220:00:222
306731230,1sleep20-21swapper/219:50:212
291398230,1sleep00-21swapper/019:38:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional