You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 02:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sun Nov 23, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3376739913350,1330cyclictest3443582sleep307:15:013
3376739913080,1305cyclictest5878662sleep310:09:553
3376739912800,1277cyclictest7090052sleep311:25:013
3376739910401,1034cyclictest652109-21sleep10:50:003
337673992724,1cyclictest0-21swapper/311:10:003
337673992118,1cyclictest0-21swapper/308:15:183
337673992118,1cyclictest0-21swapper/307:15:023
337673992017,1cyclictest0-21swapper/309:55:243
337673991917,1cyclictest0-21swapper/309:25:243
337673991916,1cyclictest0-21swapper/311:00:003
337673991916,1cyclictest0-21swapper/310:50:153
337673991815,1cyclictest0-21swapper/308:35:013
33767399176,9cyclictest364877-21turbostat07:34:553
337673991715,1cyclictest0-21swapper/312:10:003
337673991715,1cyclictest0-21swapper/307:30:013
337673991714,1cyclictest0-21swapper/310:30:163
337673991714,1cyclictest0-21swapper/310:20:173
337673991714,1cyclictest0-21swapper/309:25:003
337673991714,1cyclictest0-21swapper/308:40:013
337673991714,1cyclictest0-21swapper/308:40:003
337673991714,1cyclictest0-21swapper/307:45:003
33767399164,9cyclictest741023-21turbostat11:49:553
337673991614,1cyclictest0-21swapper/312:20:153
337673991614,1cyclictest0-21swapper/312:10:263
337673991614,1cyclictest0-21swapper/308:45:003
337673991613,1cyclictest0-21swapper/312:35:263
337673991613,1cyclictest0-21swapper/308:30:003
337673991613,1cyclictest0-21swapper/308:05:003
33767399161,12cyclictest555695-21turbostat09:49:593
7901302156,6sleep0790389-21cut12:15:160
33767399155,7cyclictest579594-21taskset10:04:183
33767399155,7cyclictest4851852sleep309:05:143
337673991514,0cyclictest0-21swapper/307:40:013
337673991513,1cyclictest0-21swapper/311:55:253
337673991513,1cyclictest0-21swapper/311:41:523
337673991513,1cyclictest0-21swapper/311:15:203
337673991513,1cyclictest0-21swapper/308:20:183
337673991512,1cyclictest0-21swapper/312:25:123
337673991512,1cyclictest0-21swapper/312:04:583
337673991512,1cyclictest0-21swapper/311:00:263
337673991512,1cyclictest0-21swapper/310:15:163
337673991512,1cyclictest0-21swapper/309:35:123
337673991512,1cyclictest0-21swapper/309:20:013
337673991512,1cyclictest0-21swapper/308:50:163
5798142140,5sleep0579822-21ssh10:04:580
5634872141,2sleep20-21swapper/209:54:292
3805952140,1sleep0382749-21kthreadcore07:40:210
337673991413,0cyclictest0-21swapper/309:54:063
337673991412,1cyclictest0-21swapper/311:10:183
337673991412,1cyclictest0-21swapper/310:13:313
337673991412,1cyclictest0-21swapper/309:04:013
337673991412,0cyclictest0-21swapper/311:21:483
337673991412,0cyclictest0-21swapper/308:06:323
337673991412,0cyclictest0-21swapper/307:50:163
337673991411,1cyclictest0-21swapper/311:35:013
337673991411,1cyclictest0-21swapper/309:10:133
337673991411,1cyclictest0-21swapper/307:45:223
3372792140,3sleep30-21swapper/307:06:453
190898140,10rtkit-daemon364099-21sendmail_mailst07:25:210
33767399135,6cyclictest8197612sleep312:30:423
33767399135,6cyclictest4606992sleep308:45:173
33767399132,8cyclictest539870-21taskset09:40:023
33767399131,9cyclictest352577-21taskset07:20:063
337673991312,0cyclictest0-21swapper/311:50:083
337673991311,1cyclictest0-21swapper/310:40:133
337673991310,1cyclictest0-21swapper/312:15:153
337673991310,1cyclictest0-21swapper/311:35:153
337673991310,1cyclictest0-21swapper/308:10:203
33767399130,10cyclictest612226-21taskset10:25:023
3367192132,6sleep0336844-21python307:05:220
5399732121,8sleep0540027-21/usr/sbin/munin09:40:120
4988242121,8sleep2498832-21ssh09:14:132
33767399121,9cyclictest530383-21taskset09:32:063
33767399121,9cyclictest477604-21taskset08:55:183
337673991210,1cyclictest0-21swapper/310:35:293
190898120,8rtkit-daemon611456-21ssh10:23:470
190898120,6rtkit-daemon675423-21ssh11:02:220
5699852111,3sleep2569993-21ssh09:55:422
5232962111,7sleep2523326-21munin-run09:30:002
5156422112,5sleep1515661-21apt-get09:25:141
33767399119,1cyclictest0-21swapper/307:55:243
190898110,8rtkit-daemon473-21gmain07:45:190
190898110,8rtkit-daemon344055-21munin-node10:50:440
190898110,7rtkit-daemon725743-21interrupts11:35:160
8219592101,7sleep1821928-21/usr/sbin/munin12:35:141
8030092100,1sleep1803098-21/usr/sbin/munin12:20:221
7805032101,6sleep0780511-21ssh12:08:030
7670582100,8sleep0768767-21cut12:00:180
6374252101,6sleep2638546-21cut10:40:172
5962472101,2sleep00-21swapper/010:15:150
5762602100,7sleep2577181-21memory10:00:182
4097022102,4sleep0409735-21apt-get08:05:130
190898100,6rtkit-daemon812241-21ssh12:27:080
190898100,5rtkit-daemon676885-21apt-get11:05:120
190898100,5rtkit-daemon642411-21ssh10:41:040
190898100,3rtkit-daemon674-21lldpd11:54:112
190898100,3rtkit-daemon619421-21ssh10:28:360
821275291,2sleep20-21swapper/212:34:172
813483291,4sleep1813589-21sed12:30:011
707413291,5sleep1707421-21sh11:21:551
589200291,5sleep1590230-21kthreadcore10:10:201
564498290,7sleep1565090-21/usr/sbin/munin09:55:161
504755290,7sleep1499464-21/usr/sbin/munin09:15:211
403426291,4sleep0403459-21apt-get08:00:110
397145291,5sleep1397294-21df_inode07:55:151
377714291,5sleep0377720-21cron07:39:590
364739291,6sleep1667-21snmpd07:28:361
343860290,2sleep2343861-21sshd07:14:182
19089890,5rtkit-daemon1928-21sendmail-mta08:11:462
19089890,4rtkit-daemon828615-21ssh12:37:370
19089890,4rtkit-daemon690308-21python311:10:220
19089890,4rtkit-daemon666921-21ssh10:56:230
19089890,4rtkit-daemon629844-21grep10:35:170
19089890,4rtkit-daemon570421-21ssh09:56:440
19089890,4rtkit-daemon547972-21ssh09:45:100
19089890,3rtkit-daemon757629-21ssh11:55:132
19089890,1rtkit-daemon0-21swapper/012:32:020
816237280,5sleep1818276-21kthreadcore12:30:201
811154280,5sleep2811273-21switchtime12:25:202
786936280,5sleep0787047-21mailstats12:10:240
773627281,3sleep20-21swapper/212:05:122
741801282,3sleep1742581-21sh11:45:171
730535281,4sleep20-21swapper/211:35:232
716807280,2sleep2716815-21ssh11:29:542
700052280,1sleep00-21swapper/011:18:380
668753281,3sleep1668794-21apt-get11:00:121
659918280,6sleep2659926-21ssh10:54:282
658317280,1sleep10-21swapper/110:50:391
601998280,1sleep10-21swapper/110:15:271
594728280,2sleep2594736-21ssh10:12:132
581799281,5sleep0583767-21kthreadcore10:05:170
422289281,3sleep0422311-21apt-get08:15:130
415980281,3sleep0416013-21apt-get08:10:130
377994281,3sleep2378027-21apt-get07:40:142
371346281,4sleep1371390-21munin-run07:35:001
352981280,6sleep20-21swapper/207:20:142
346295281,4sleep2346385-21grep07:15:132
19089880,4rtkit-daemon754610-21ntp_states11:50:210
19089880,4rtkit-daemon741543-21diskstats11:45:140
19089880,4rtkit-daemon740191-21ssh11:43:070
19089880,4rtkit-daemon651779-21ssh10:49:100
19089880,4rtkit-daemon539175-21ssh09:38:580
19089880,4rtkit-daemon438-21wpa_supplicant07:50:180
19089880,4rtkit-daemon415722-21cron08:10:012
19089880,4rtkit-daemon346451-21diskstats07:15:150
19089880,4rtkit-daemon344055-21munin-node09:02:032
19089880,4rtkit-daemon344055-21munin-node08:26:372
19089880,4rtkit-daemon299-21systemd-journal11:33:380
19089880,3rtkit-daemon800837-21kthreadcore12:20:170
19089880,3rtkit-daemon299-21systemd-journal09:51:370
19089880,1rtkit-daemon0-21swapper/011:57:580
19089880,1rtkit-daemon0-21swapper/007:56:070
823429270,4sleep2825361-21cut12:35:192
797966270,4sleep2798173-21/usr/sbin/munin12:20:152
738854271,3sleep2739062-21users11:40:242
723333270,4sleep1723341-21ssh11:31:181
709075271,3sleep0709112-21apt-get11:25:110
702375270,5sleep2703451-21kthreadcore11:20:162
651081271,4sleep2651089-21ssh10:47:302
587886271,3sleep2587906-21munin-run10:09:592
586610270,5sleep1586616-21ssh10:06:581
579036270,4sleep1579040-21ssh10:03:061
549989270,4sleep2552001-21grep09:45:202
546870270,1sleep20-21swapper/209:43:112
541685270,4sleep1543660-21tr09:40:171
524366270,4sleep2525166-21sed09:30:172
504671271,3sleep0504754-21ntp_states09:15:200
499136270,4sleep1499321-21sed09:15:001
497666270,4sleep0497674-21ssh09:11:270
466326271,3sleep0466360-21apt-get08:50:110
434642270,5sleep1434688-21latency_hist08:25:011
373440270,5sleep2375483-21kthreadcore07:35:172
357907271,3sleep10-21swapper/107:20:211
343919270,4sleep1343946-21awk07:15:001
340493270,5sleep0342655-21kthreadcore07:10:190
335785271,3sleep10-21swapper/107:05:191
19089870,4rtkit-daemon421-21avahi-daemon08:55:442
19089870,4rtkit-daemon344055-21munin-node08:40:402
19089870,4rtkit-daemon344055-21munin-node08:38:552
19089870,4rtkit-daemon344055-21munin-node08:38:552
19089870,4rtkit-daemon344055-21munin-node08:16:052
19089870,4rtkit-daemon1928-21sendmail-mta07:20:310
19089870,3rtkit-daemon674-21lldpd09:15:092
19089870,3rtkit-daemon674-21lldpd07:34:220
19089870,3rtkit-daemon671-21ntpd09:22:332
19089870,3rtkit-daemon667-21snmpd10:11:070
19089870,3rtkit-daemon667-21snmpd08:45:392
19089870,3rtkit-daemon621377-21sed10:30:150
19089870,3rtkit-daemon531066-21ssh09:33:420
19089870,3rtkit-daemon344055-21munin-node11:21:180
19089870,3rtkit-daemon344055-21munin-node08:31:382
19089870,3rtkit-daemon1928-21sendmail-mta08:50:162
19089870,3rtkit-daemon1928-21sendmail-mta08:22:012
19089870,1rtkit-daemon0-21swapper/209:08:132
791306261,2sleep20-21swapper/212:15:182
789742263,1sleep10-21swapper/112:15:131
780646260,1sleep10-21swapper/112:08:281
771770260,1sleep20-21swapper/212:01:282
749352261,3sleep1749389-21apt-get11:50:131
724637260,1sleep2724645-21ssh11:34:202
698723260,3sleep2698729-21ssh11:15:282
690953260,3sleep2690961-21ssh11:11:112
683609261,3sleep1667-21snmpd11:07:461
676312260,1sleep20-21swapper/211:04:332
667588260,1sleep10-21swapper/110:57:591
643257260,1sleep10-21swapper/110:43:051
634213260,3sleep1634221-21ssh10:35:351
628773260,3sleep2628977-21irqcore10:35:172
627405262,2sleep1331ktimers/110:33:351
610997260,3sleep1611005-21ssh10:22:411
604301261,2sleep20-21swapper/210:20:132
562294261,3sleep1562300-21ssh09:51:351
548034260,3sleep1548112-21grep09:45:151
532710261,3sleep1533567-21cut09:35:161
513535260,4sleep1513543-21ssh09:20:551
486970260,4sleep0489010-21kthreadcore09:05:190
448910263,1sleep0221rcuc/008:35:170
448910263,1sleep0221rcuc/008:35:170
441178260,3sleep0441211-21apt-get08:30:120
435151260,4sleep00-21swapper/008:25:150
434437260,1sleep00-21swapper/008:22:240
428149260,1sleep10-21swapper/108:17:511
408629260,3sleep2408754-21python308:00:212
404671260,4sleep1405813-21kthreadcore08:00:161
396586260,3sleep2396908-21latency_hist07:55:012
372582260,3sleep1373439-21cut07:35:161
364480260,1sleep20-21swapper/207:25:242
331598260,2sleep20-21swapper/207:05:122
786714250,2sleep20-21swapper/212:10:222
763268250,1sleep10-21swapper/111:55:291
748376251,2sleep20-21swapper/211:48:212
738720250,2sleep10-21swapper/111:40:231
731144250,3sleep1731152-21ssh11:35:421
690649251,2sleep10-21swapper/111:10:251
650979250,3sleep1650987-21ssh10:47:171
626741250,2sleep2626749-21ssh10:31:512
618408250,3sleep20-21swapper/210:26:152
612865251,2sleep10-21swapper/110:25:181
533568250,3sleep2535575-21kthreadcore09:35:172
515585250,1sleep00-21swapper/009:25:050
513458250,1sleep00-21swapper/009:20:420
484704250,1sleep00-21swapper/009:02:330
484129251,2sleep10-21swapper/109:00:221
472626251,1sleep0472659-21apt-get08:55:130
463595250,3sleep1465036-21memory08:45:191
461582250,2sleep00-21swapper/008:45:180
459478250,1sleep00-21swapper/008:41:350
453205250,1sleep10-21swapper/108:36:541
453205250,1sleep10-21swapper/108:36:541
442635250,3sleep1444237-21cat08:30:181
435341250,2sleep1436260-21sed08:25:161
416961251,2sleep10-21swapper/108:10:181
414882250,3sleep10-21swapper/108:05:221
402866250,1sleep20-21swapper/207:56:242
384304250,3sleep1384353-21cpuspeed_turbos07:45:131
371161250,1sleep20-21swapper/207:32:402
346259250,2sleep10-21swapper/107:15:121
770878240,1sleep10-21swapper/112:00:221
710560240,2sleep1710603-21kthreadcore11:25:161
698263240,2sleep1698384-21python311:15:221
683618240,2sleep2674880-21kworker/u16:0+rpciod11:07:472
667144240,3sleep20-21swapper/210:56:562
602041240,2sleep2602049-21ssh10:15:322
530951240,3sleep10-21swapper/109:33:291
491024240,3sleep10-21swapper/109:07:521
474083240,2sleep1475726-21kthreadcore08:55:171
386052240,2sleep20-21swapper/207:45:182
781311230,1sleep10-21swapper/112:10:011
466363230,1sleep10-21swapper/108:50:121
459055230,1sleep10-21swapper/108:40:241
377743230,1sleep10-21swapper/107:40:011
390533220,1sleep10-21swapper/107:50:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional