You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 04:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Wed Nov 19, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2482749911111,1106cyclictest515677-21sleep22:20:003
248274993831,379cyclictest364219-21turbostat20:44:563
248274993700,367cyclictest395680-21turbostat21:09:553
248274992219,1cyclictest0-21swapper/319:45:003
24827499208,8cyclictest7106112sleep300:20:143
24827499196,10cyclictest613100-21taskset23:20:023
248274991815,1cyclictest0-21swapper/321:59:593
2577912170,6sleep2421rcuc/219:15:192
248274991714,1cyclictest0-21swapper/300:16:443
248274991613,1cyclictest0-21swapper/321:45:013
248274991613,1cyclictest0-21swapper/319:16:353
248274991613,1cyclictest0-21swapper/300:00:013
24827499155,6cyclictest6843242sleep300:00:543
248274991513,1cyclictest0-21swapper/319:54:293
248274991513,1cyclictest0-21swapper/319:49:153
248274991512,1cyclictest0-21swapper/323:45:243
248274991512,1cyclictest0-21swapper/322:35:003
248274991512,1cyclictest0-21swapper/320:03:013
248274991512,1cyclictest0-21swapper/300:30:213
24827499151,11cyclictest499475-21turbostat22:10:003
24827499151,11cyclictest426705-21taskset21:25:013
24827499151,10cyclictest623003-21taskset23:25:183
24827499150,11cyclictest702220-21turbostat00:15:003
24827499145,7cyclictest571506-21taskset22:53:093
24827499145,7cyclictest571506-21taskset22:53:083
24827499144,7cyclictest375892-21taskset20:45:233
248274991412,1cyclictest0-21swapper/322:46:213
248274991412,1cyclictest0-21swapper/321:55:003
248274991412,1cyclictest0-21swapper/320:40:003
248274991412,1cyclictest0-21swapper/320:20:463
248274991412,0cyclictest0-21swapper/321:23:113
248274991411,1cyclictest0-21swapper/323:35:273
248274991411,1cyclictest0-21swapper/322:39:003
248274991411,1cyclictest0-21swapper/322:03:373
248274991411,1cyclictest0-21swapper/320:53:533
248274991411,1cyclictest0-21swapper/319:56:353
248274991411,1cyclictest0-21swapper/300:05:213
24827499141,10cyclictest592549-21chrt23:05:193
24827499141,10cyclictest580703-21taskset23:00:013
24827499141,10cyclictest383686-21taskset20:55:163
24827499141,10cyclictest284383-21taskset19:35:193
24827499140,11cyclictest604910-21taskset23:15:003
7038122130,4sleep2704083-21kthreadcore00:15:202
4896462130,6sleep1489654-21ssh22:01:381
24827499134,6cyclictest7407412sleep300:35:283
24827499131,9cyclictest662008-21taskset23:50:063
24827499131,9cyclictest572634-21taskset22:55:093
24827499131,9cyclictest410278-21taskset21:15:033
24827499131,9cyclictest351916-21chrt20:30:123
24827499131,8cyclictest6458512sleep323:40:153
248274991311,1cyclictest0-21swapper/319:32:553
24827499131,10cyclictest637109-21turbostat23:35:013
24827499131,10cyclictest531359-21taskset22:29:003
24827499131,10cyclictest442611-21turbostat21:35:003
248274991310,1cyclictest0-21swapper/322:10:213
248274991310,1cyclictest0-21swapper/321:45:203
24827499130,10cyclictest275313-21chrt19:27:583
5212672120,1sleep30-21swapper/322:20:243
24827499129,1cyclictest0-21swapper/322:41:083
24827499121,9cyclictest724638-21chrt00:25:343
24827499121,9cyclictest612293-21taskset23:18:463
24827499121,9cyclictest326216-21taskset20:06:163
24827499121,8cyclictest446751-21taskset21:35:193
24827499121,8cyclictest267681-21taskset19:20:213
248274991210,1cyclictest0-21swapper/321:00:143
248274991210,1cyclictest0-21swapper/320:25:383
24827499120,9cyclictest402183-21taskset21:10:073
24827499120,9cyclictest338733-21taskset20:15:503
190898120,3rtkit-daemon254614-21munin-node21:29:080
4921982111,7sleep0493145-21cut22:05:190
4733382110,8sleep2473346-21ssh21:51:342
4501652110,2sleep1171rcu_preempt21:38:471
24827499112,7cyclictest331949-21taskset20:10:243
24827499110,4cyclictest0-21swapper/319:10:023
190898110,6rtkit-daemon475364-21df_inode21:55:140
190898110,4rtkit-daemon515165-21ssh22:18:540
190898110,4rtkit-daemon401896-21diskmemload00:17:401
6617152100,7sleep0661961-21date23:50:010
2472222101,6sleep0247294-21ntp_states19:05:240
190898100,7rtkit-daemon229200-21kworker/u16:3+events_freezable_pwr_efficient19:25:481
190898100,6rtkit-daemon696123-21cut00:10:190
190898100,6rtkit-daemon319-21systemd-udevd20:21:381
190898100,5rtkit-daemon639527-21kthreadcore23:35:200
190898100,5rtkit-daemon438-21wpa_supplicant00:29:302
190898100,4rtkit-daemon254614-21munin-node23:08:570
190898100,3rtkit-daemon254614-21munin-node21:23:040
190898100,1rtkit-daemon0-21swapper/023:59:090
677932291,5sleep2677981-40updatedb.mlocat00:00:012
670144291,4sleep1670181-21apt-get23:55:141
653936291,4sleep1653970-21apt-get23:45:131
651051291,4sleep0651115-21ntp_states23:40:230
635203290,2sleep1635211-21ssh23:30:391
628301290,6sleep2628309-21ssh23:28:182
621213290,7sleep072-21kcompactd023:25:090
546387291,6sleep1546395-21ssh22:36:171
507904291,4sleep2507939-21apt-get22:15:142
491106291,5sleep2491361-21latency_hist22:05:002
465628291,4sleep2465636-21ssh21:47:162
442047290,2sleep20-21swapper/221:33:462
370483291,6sleep01223-21cron20:45:000
320512291,4sleep0320545-21apt-get20:05:140
307423291,5sleep0307439-21munin-run19:55:000
295145290,2sleep20-21swapper/219:45:162
295120291,4sleep1295142-21apt-get19:45:151
19089890,6rtkit-daemon401896-21diskmemload22:21:200
19089890,6rtkit-daemon263319-21df_inode19:20:151
19089890,5rtkit-daemon588377-21ssh23:04:370
19089890,5rtkit-daemon438-21wpa_supplicant22:50:142
19089890,5rtkit-daemon438-21wpa_supplicant22:50:132
19089890,5rtkit-daemon319-21systemd-udevd22:25:370
19089890,5rtkit-daemon319-21systemd-udevd19:15:381
19089890,5rtkit-daemon254614-21munin-node22:10:160
19089890,5rtkit-daemon0-21swapper/021:16:410
19089890,4rtkit-daemon693023-21ssh00:07:220
19089890,4rtkit-daemon684960-21ssh00:02:220
19089890,4rtkit-daemon669058-21ssh23:53:190
19089890,4rtkit-daemon636451-21ssh23:33:320
19089890,4rtkit-daemon612082-21ssh23:18:180
19089890,4rtkit-daemon473770-21ssh21:52:340
19089890,4rtkit-daemon440753-21ssh21:30:380
19089890,1rtkit-daemon0-21swapper/022:04:140
726732281,4sleep1726552-21apt-get00:30:011
726534281,4sleep0726727-21apt-config00:30:000
710648281,4sleep0710803-21df_inode00:20:140
709886280,1sleep00-21swapper/000:19:060
701774280,1sleep2701782-21ssh00:14:002
684410280,2sleep2684418-21ssh00:01:052
651723280,5sleep1651731-21ssh23:40:401
643797280,1sleep10-21swapper/123:36:041
564583281,4sleep0564741-21df_inode22:50:150
564583281,4sleep0564741-21df_inode22:50:140
548368281,3sleep2548405-21apt-get22:40:152
532178280,5sleep1532131-21/usr/sbin/munin22:30:161
465210280,5sleep1465218-21ssh21:46:151
364868281,4sleep1365434-21cut20:40:201
326962281,4sleep0327163-21hddtemp_smartct20:10:190
315126280,5sleep0316087-21grep20:00:200
308336281,4sleep0308514-21cut19:55:160
306611281,4sleep1306736-21python319:50:251
294849280,5sleep2295083-21cat19:45:012
280915281,4sleep2281040-21python319:30:252
269467281,4sleep0269622-21diskstats19:25:150
19089880,5rtkit-daemon401896-21diskmemload22:58:170
19089880,5rtkit-daemon1928-21sendmail-mta20:57:430
19089880,4rtkit-daemon671-21ntpd20:06:171
19089880,4rtkit-daemon667-21snmpd20:30:541
19089880,4rtkit-daemon438-21wpa_supplicant21:03:190
19089880,4rtkit-daemon438-21wpa_supplicant20:27:361
19089880,4rtkit-daemon319506-21mailstats20:00:251
19089880,4rtkit-daemon299-21systemd-journal20:15:381
19089880,3rtkit-daemon671-21ntpd21:43:030
19089880,1rtkit-daemon1907-21rtkit-daemon22:46:442
19089880,1rtkit-daemon0-21swapper/100:20:161
19089880,1rtkit-daemon0-21swapper/023:24:430
728160270,4sleep2728677-21kthreadcore00:30:172
716553270,5sleep2716559-21ssh00:20:482
693329270,4sleep2693336-21sh00:08:012
644741270,1sleep20-21swapper/223:38:172
620440271,4sleep1620448-21ssh23:23:541
605984270,4sleep1606845-21grep23:15:191
598486270,5sleep1600173-21kthreadcore23:10:271
586816270,1sleep10-21swapper/123:00:561
562410270,1sleep00-21swapper/022:45:540
550613270,5sleep0552681-21kthreadcore22:40:200
546299270,1sleep20-21swapper/222:36:062
529505270,4sleep2529622-21awk22:25:232
523100271,4sleep20-21swapper/222:23:362
505846270,3sleep20-21swapper/222:10:572
472910270,4sleep1472918-21ssh21:50:291
459067271,2sleep00-21swapper/021:45:150
443335270,5sleep0443916-21sed21:35:160
436955270,5sleep1439022-21kthreadcore21:30:211
417883270,5sleep2417891-21ssh21:19:322
357037271,3sleep0357093-21ntp_states20:30:220
345067270,1sleep00-21swapper/020:20:570
339062270,3sleep0339106-21awk20:19:590
327164271,3sleep10-21swapper/120:10:201
307462270,4sleep1307437-21apt-get19:55:011
287596270,4sleep0287739-21wc19:35:240
275734271,3sleep0275767-21apt-get19:30:160
263545271,3sleep00-21swapper/019:20:180
19089870,4rtkit-daemon674-21lldpd20:35:111
19089870,4rtkit-daemon671-21ntpd21:10:370
19089870,4rtkit-daemon473-21gmain19:10:301
19089870,4rtkit-daemon254614-21munin-node19:40:061
19089870,3rtkit-daemon81364-21packagekitd19:05:191
19089870,3rtkit-daemon701238-21ssh00:12:421
19089870,3rtkit-daemon671-21ntpd20:56:251
19089870,3rtkit-daemon667-21snmpd20:50:261
19089870,3rtkit-daemon667-21snmpd19:33:551
19089870,3rtkit-daemon254614-21munin-node21:06:080
19089870,3rtkit-daemon254614-21munin-node20:49:281
19089870,3rtkit-daemon254614-21munin-node00:36:022
19089870,1rtkit-daemon0-21swapper/119:38:151
734329260,4sleep00-21swapper/000:33:450
684496260,1sleep10-21swapper/100:01:161
667822260,1sleep10-21swapper/123:50:301
632580260,3sleep2634432-21cat23:30:202
621193261,3sleep1621201-21ssh23:25:061
620517260,1sleep20-21swapper/223:24:032
611832260,1sleep20-21swapper/223:17:412
590491260,3sleep2592547-21kthreadcore23:05:192
588873261,3sleep1588881-21ssh23:05:101
586603260,1sleep20-21swapper/223:00:292
572307261,3sleep1299-21systemd-journal22:55:001
572307261,3sleep1299-21systemd-journal22:55:001
538063260,1sleep00-21swapper/022:30:430
535423260,3sleep2537216-21sed22:30:212
516442260,3sleep1517298-21sed22:20:181
498358260,1sleep20-21swapper/222:07:292
481277260,1sleep10-21swapper/121:56:021
481004260,1sleep20-21swapper/221:55:272
458272260,1sleep20-21swapper/221:43:572
433987260,1sleep20-21swapper/221:28:462
421448260,3sleep1423610-21cut21:20:191
408469260,4sleep10-21swapper/121:11:301
403760261,2sleep20-21swapper/221:10:192
389336261,2sleep20-21swapper/221:00:012
376697260,1sleep20-21swapper/220:49:312
309441262,2sleep244-21ksoftirqd/219:55:182
295503260,4sleep0295618-21irqcore19:45:190
294005260,3sleep0294080-21ps19:40:220
253600261,2sleep00-21swapper/019:10:210
242163260,3sleep30-21swapper/319:05:023
19089860,2rtkit-daemon602749-21unixbench_singl23:10:340
740514250,2sleep10-21swapper/100:35:251
726474250,3sleep1726480-21ssh00:29:531
686363250,2sleep10-21swapper/100:05:041
667457250,3sleep2667686-21awk23:50:232
649621250,3sleep2650907-21memory23:40:212
604701251,2sleep20-21swapper/223:15:012
579857250,3sleep2579865-21ssh22:58:422
579122251,2sleep10-21swapper/122:57:011
562417250,2sleep1562425-21ssh22:45:551
515341250,3sleep10-21swapper/122:19:181
505669250,1sleep10-21swapper/122:10:371
497973251,2sleep10-21swapper/122:06:321
452567250,3sleep1454431-21kthreadcore21:40:191
434407250,3sleep1434415-21ssh21:29:451
410364250,2sleep10-21swapper/121:15:151
398357251,2sleep10-21swapper/121:05:201
395628251,2sleep1667-21snmpd21:04:411
395391250,1sleep20-21swapper/221:01:372
382945250,1sleep00-21swapper/020:53:500
365436251,2sleep20-21swapper/220:40:212
364091250,1sleep20-21swapper/220:38:502
345633250,1sleep00-21swapper/020:25:090
340323251,2sleep20-21swapper/220:20:182
338211250,3sleep2338260-21ps20:15:262
319565250,2sleep20-21swapper/220:00:262
301195250,3sleep20-21swapper/219:50:022
283245250,2sleep2284382-21grep19:35:192
269429250,2sleep20-21swapper/219:25:132
263313250,2sleep2263359-21cut19:20:172
242373250,2sleep20-21swapper/219:05:172
742036240,2sleep00-21swapper/000:37:530
653909240,2sleep2667-21snmpd23:45:102
554219240,2sleep1554227-21ssh22:40:371
541767240,2sleep00-21swapper/022:35:200
530066240,3sleep1523547-21kworker/u16:2+nfsiod22:25:581
448962240,2sleep20-21swapper/221:35:592
426267240,2sleep20-21swapper/221:24:362
400515240,2sleep20-21swapper/221:05:212
377868240,2sleep20-21swapper/220:50:182
376533240,3sleep00-21swapper/020:47:250
357931240,1sleep20-21swapper/220:35:012
327476240,1sleep20-21swapper/220:10:202
326508240,2sleep2326707-21latency_hist20:10:012
248670240,2sleep20-21swapper/219:10:152
358180230,1sleep00-21swapper/020:35:070
351398230,1sleep20-21swapper/220:26:432
254470230,1sleep00-21swapper/019:15:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional