You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 20:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Dec 27, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52181922580,1sleep0521305-21apt-get09:35:020
366377212655,66sleep30-21swapper/307:09:123
36638628756,10sleep20-21swapper/207:09:192
36625728165,10sleep10-21swapper/107:07:381
36613928061,13sleep00-21swapper/007:06:090
5706532280,2sleep00-21swapper/010:20:190
366568992219,1cyclictest0-21swapper/308:30:013
366568992118,1cyclictest0-21swapper/310:50:183
366568992118,1cyclictest0-21swapper/308:35:003
366568992118,1cyclictest0-21swapper/307:45:273
366568992117,2cyclictest0-21swapper/307:38:303
6332472201,5sleep1633316-21irqstats11:20:171
366568992017,1cyclictest0-21swapper/311:00:163
366568992017,1cyclictest0-21swapper/310:48:293
366568992016,2cyclictest0-21swapper/308:05:003
366568992013,3cyclictest6801202sleep312:05:133
36656899201,15cyclictest0-21swapper/307:40:113
366568991917,1cyclictest0-21swapper/311:50:133
366568991916,1cyclictest0-21swapper/312:35:163
36656899191,3cyclictest494809-21turbostat09:09:593
36656899191,15cyclictest566752-21taskset10:15:173
36656899191,14cyclictest0-21swapper/312:15:133
36656899191,13cyclictest0-21swapper/312:25:103
36656899191,13cyclictest0-21swapper/307:50:123
36656899189,6cyclictest595476-21turbostat10:45:013
366568991816,1cyclictest0-21swapper/311:55:143
366568991815,1cyclictest0-21swapper/311:35:173
366568991815,1cyclictest0-21swapper/311:15:193
366568991815,1cyclictest0-21swapper/310:15:003
366568991815,1cyclictest0-21swapper/310:00:273
366568991815,1cyclictest0-21swapper/309:45:173
366568991815,1cyclictest0-21swapper/309:10:263
366568991815,1cyclictest0-21swapper/308:05:143
5735012171,12sleep1573704-21sed10:20:291
5045052171,5sleep1504519-21readlink09:15:241
366568991715,1cyclictest0-21swapper/310:25:003
366568991714,1cyclictest0-21swapper/309:35:013
366568991713,2cyclictest0-21swapper/311:40:153
366568991712,3cyclictest0-21swapper/308:40:113
36656899171,13cyclictest520605-21taskset09:30:253
36656899171,13cyclictest0-21swapper/311:50:013
36656899171,13cyclictest0-21swapper/308:50:113
36656899171,13cyclictest0-21swapper/308:35:113
36656899171,13cyclictest0-21swapper/307:20:003
36656899171,12cyclictest0-21swapper/311:05:113
36656899171,12cyclictest0-21swapper/309:50:123
36656899171,12cyclictest0-21swapper/307:55:183
36656899171,10cyclictest690291-21taskset12:14:333
36656899171,10cyclictest516084-21taskset09:30:003
36656899170,14cyclictest0-21swapper/311:30:003
36656899170,14cyclictest0-21swapper/308:10:273
36656899170,12cyclictest0-21swapper/312:00:113
36656899170,10cyclictest637369-21taskset11:20:503
6580182161,10sleep0658073-21ntp_offset11:40:200
6395142160,11sleep1641252-21tr11:25:191
4796292161,4sleep1479841-21hddtemp_smartct08:55:161
3670662161,10sleep1367283-21awk07:10:151
36656899161,9cyclictest606474-21taskset10:55:093
36656899161,9cyclictest542769-21taskset09:55:093
36656899161,3cyclictest0-21swapper/310:25:123
36656899161,3cyclictest0-21swapper/310:25:123
366568991613,1cyclictest0-21swapper/309:00:003
366568991612,2cyclictest0-21swapper/310:30:223
36656899161,12cyclictest0-21swapper/308:45:203
36656899161,12cyclictest0-21swapper/308:15:133
36656899161,11cyclictest699862-21chrt12:20:223
36656899161,11cyclictest531152-21chrt09:40:233
36656899161,10cyclictest646143-21taskset11:30:163
36656899161,10cyclictest505117-21taskset09:18:343
36656899160,14cyclictest0-21swapper/307:20:223
36656899160,13cyclictest0-21swapper/310:35:263
36656899160,13cyclictest0-21swapper/307:30:163
36656899160,10cyclictest626923-21taskset11:10:323
6964052151,9sleep2697763-21cut12:20:182
6696522151,4sleep0669671-21apt-get11:55:110
6278092151,2sleep10-21swapper/111:15:171
5826572151,4sleep0583788-21memory10:30:180
4586282151,10sleep1458688-21cut08:35:121
36656899151,9cyclictest711238-21taskset12:31:093
36656899151,9cyclictest711238-21taskset12:31:093
36656899151,9cyclictest389404-21taskset07:27:353
36656899151,2cyclictest0-21swapper/308:20:193
366568991512,1cyclictest0-21swapper/310:05:203
366568991512,1cyclictest0-21swapper/307:10:183
36656899150,12cyclictest0-21swapper/309:20:113
6736352141,8sleep1673689-21ntpq11:55:201
6067712141,4sleep0606981-21hddtemp_smartct10:55:160
5962942141,8sleep1596526-21cat10:45:141
366568991411,1cyclictest0-21swapper/309:00:203
7071332134,5sleep0708034-21grep12:30:150
7071332134,5sleep0708034-21grep12:30:140
7018872131,3sleep10-21swapper/112:25:141
6859042131,5sleep10-21swapper/112:10:151
5910922131,4sleep0592527-21grep10:40:160
5375832131,7sleep1537602-21apt-get09:50:111
5014042131,4sleep2503100-21grep09:15:172
4726982130,9sleep2472787-21ntpq08:45:192
6864852121,6sleep2688157-21cut12:10:162
6382432121,7sleep2638400-21tail11:25:152
6053402121,7sleep0605465-21wc10:50:220
5999832122,4sleep2600016-21ntpq10:45:192
5892122120,7sleep1589258-21ntpq10:35:201
5602832120,8sleep0562050-21kthreadcore10:10:180
5480502121,7sleep0548148-21cpuspeed_turbos10:00:120
3828492121,7sleep0383304-21uname07:20:200
3752732121,7sleep0376958-21kthreadcore07:15:190
3710642121,7sleep0371224-21timerwakeupswit07:10:260
6850922111,5sleep1685120-21munin-run12:09:591
5799732112,4sleep2579992-21apt-get10:30:132
5212952111,5sleep1521313-21munin-run09:35:001
4947732110,7sleep2494909-21date09:10:002
4833962111,4sleep0483449-21ntp_states08:55:230
4364582114,3sleep1436572-21cat08:10:271
4109492111,4sleep2410968-21apt-get07:50:122
3901472111,5sleep0390356-21hddtemp_smartct07:30:140
3867632111,6sleep1388404-21latency07:25:181
7157772101,4sleep0715818-21ntpq12:35:200
6117362101,4sleep1611755-21apt-get11:00:131
6017832104,3sleep10-21swapper/110:50:151
5892602101,5sleep2589314-21ntpq10:35:212
5498412101,5sleep1551598-21tr10:00:171
5415242100,2sleep20-21swapper/209:50:202
5363282101,4sleep0536376-21ntpq09:45:210
5326642101,4sleep2532873-21hddtemp_smartct09:45:152
5094702101,5sleep0509516-21ld09:20:200
4955302101,4sleep1495613-21irqstats09:10:161
4639012101,4sleep2464110-21hddtemp_smartct08:40:142
4426922101,4sleep0442711-21apt-get08:20:120
4416812101,5sleep1441686-21/usr/sbin/munin08:15:241
4307052101,4sleep2430761-21ntp_states08:05:242
705145291,4sleep0705285-21/usr/sbin/munin12:25:170
690889291,4sleep0691110-21basename12:15:150
678882291,3sleep0678929-21ntpq12:00:200
659228290,1sleep10-21swapper/111:45:111
644014290,5sleep10-21swapper/111:30:151
626143291,4sleep1626260-21ld11:10:191
606499291,4sleep2606684-21expr10:55:132
602066291,4sleep2603686-21cut10:50:162
564004291,3sleep2564260-21df10:15:132
558997291,4sleep2559068-21irqstats10:10:162
553264291,3sleep00-21swapper/010:05:140
543344291,4sleep2544591-21cut09:55:172
537879291,4sleep0538091-21smartctl09:50:140
522413291,4sleep1523890-21tr09:35:161
513685291,3sleep0514877-21memory09:25:170
492574291,3sleep0493617-21memory09:05:170
484615291,3sleep00-21swapper/009:00:130
447920290,3sleep20-21swapper/208:25:112
438457291,4sleep2440130-21kthreadcore08:15:182
420135291,3sleep0420176-21ntpq07:55:190
416836291,4sleep1418455-21grep07:55:171
399984291,4sleep2400011-21systemd07:38:292
399884290,1sleep00-21swapper/007:37:120
393761291,4sleep1393809-21ntpq07:30:191
379338291,3sleep1379357-21apt-get07:20:131
367284291,4sleep2367675-21kthreadcore07:10:162
715820281,2sleep10-21swapper/112:35:211
708036281,3sleep1709768-21kthreadcore12:30:161
708036281,3sleep1709768-21kthreadcore12:30:151
690610281,3sleep2690869-21df_inode12:15:132
666314281,3sleep2668098-21kthreadcore11:50:162
664407281,4sleep0248-21systemd-journal11:50:020
637043284,1sleep00-21swapper/011:20:280
627226281,3sleep2627228-21modprobe11:14:292
615738281,2sleep2615786-21ntpq11:00:222
585619281,3sleep0585769-21/usr/sbin/munin10:35:150
579016281,4sleep0579041-21cat10:25:250
579016281,4sleep0579041-21cat10:25:250
578612281,3sleep2578704-21ld10:25:202
578612281,3sleep2578704-21ld10:25:192
575008280,5sleep1575140-21gltestperf10:25:151
575008280,5sleep1575140-21gltestperf10:25:141
563445281,3sleep1563756-21apt-key10:15:001
553286281,3sleep1553539-21df10:05:141
520283282,2sleep2520328-21ntpq09:30:212
474444281,3sleep1474616-21if_err_enp1s008:50:151
447548280,1sleep20-21swapper/208:23:342
432789281,3sleep0434354-21kthreadcore08:10:180
432480281,4sleep244-21ksoftirqd/208:10:162
421442281,3sleep0151ktimers/008:00:120
414979281,4sleep0415031-21awk07:50:220
404563281,3sleep2404611-21/usr/sbin/munin07:40:222
404508281,3sleep1404556-21ntpq07:40:211
395090281,3sleep1395234-21cstates07:35:131
389878281,3sleep2389986-21cpuspeed_turbos07:30:122
384355280,1sleep00-21swapper/007:25:020
379909281,3sleep2381094-21kthreadcore07:20:182
695832270,2sleep10-21swapper/112:20:131
674482271,2sleep20-21swapper/211:58:312
669291270,1sleep10-21swapper/111:53:441
657873270,2sleep2657962-21ntpq11:40:182
653670270,1sleep10-21swapper/111:39:301
644471271,2sleep20-21swapper/211:30:162
641803270,2sleep00-21swapper/011:25:220
637685270,4sleep20-21swapper/211:24:552
627116270,1sleep00-21swapper/011:13:030
617260271,3sleep0617471-21hddtemp_smartct11:05:140
616646270,1sleep00-21swapper/011:04:130
568055271,3sleep0568197-21mailstats10:15:240
563982270,2sleep10-21swapper/110:15:131
548318270,4sleep20-21swapper/210:00:142
547724270,1sleep10-21swapper/109:59:451
537037271,4sleep10-21swapper/109:46:221
527120270,2sleep20-21swapper/209:40:152
516587271,3sleep0516760-21head09:30:140
515761270,1sleep10-21swapper/109:26:301
511022270,2sleep20-21swapper/209:25:112
510355270,1sleep20-21swapper/209:23:242
509517271,2sleep1509567-21ntpq09:20:221
495614271,2sleep20-21swapper/209:10:172
495035270,1sleep00-21swapper/009:10:060
494500270,1sleep10-21swapper/109:06:301
478067271,2sleep2478108-21ntpq08:50:202
473487270,4sleep10-21swapper/108:46:061
468396270,1sleep10-21swapper/108:42:571
468352270,1sleep00-21swapper/008:42:250
457991270,1sleep20-21swapper/208:33:452
454236271,2sleep0455911-21kthreadcore08:30:180
448219271,2sleep1448431-21smartctl08:25:141
426673271,3sleep1426822-21expr08:05:141
426651270,2sleep00-21swapper/008:05:130
426242270,1sleep10-21swapper/108:03:131
410630270,1sleep20-21swapper/207:49:202
406344271,3sleep1407825-21kthreadcore07:45:191
706558260,1sleep20-21swapper/212:30:042
706558260,1sleep20-21swapper/212:30:042
701359260,3sleep2421rcuc/212:25:112
700542260,1sleep00-21swapper/012:21:420
690589260,2sleep10-21swapper/112:15:121
689993260,3sleep00-21swapper/012:10:470
684867260,4sleep20-21swapper/212:07:082
680780261,2sleep00-21swapper/012:05:170
679678260,1sleep10-21swapper/112:02:421
660016261,2sleep20-21swapper/211:45:202
657316260,3sleep10-21swapper/111:40:171
653452260,1sleep00-21swapper/011:36:400
648708260,4sleep20-21swapper/211:35:062
648141260,1sleep00-21swapper/011:30:580
632198260,1sleep20-21swapper/211:16:422
628282261,2sleep00-21swapper/011:15:200
617542261,2sleep20-21swapper/211:05:152
616969260,2sleep10-21swapper/111:05:101
606477260,2sleep10-21swapper/110:55:121
598449261,2sleep0599843-21load10:45:170
595306260,1sleep20-21swapper/210:43:082
595159260,4sleep10-21swapper/110:41:131
531913260,3sleep10-21swapper/109:42:291
526774260,4sleep20-21swapper/209:39:182
500639261,2sleep00-21swapper/009:15:150
479334260,2sleep20-21swapper/208:55:132
463241260,1sleep20-21swapper/208:38:582
458345260,1sleep00-21swapper/008:35:070
457179261,2sleep10-21swapper/108:30:231
452706261,2sleep0800-21snmpd08:27:530
442714261,1sleep10-21swapper/108:20:131
415570260,1sleep10-21swapper/107:50:511
405740260,2sleep00-21swapper/007:45:130
401074261,2sleep00-21swapper/007:40:160
388408261,2sleep20-21swapper/207:25:192
378966260,4sleep20-21swapper/207:18:572
378877260,1sleep10-21swapper/107:17:481
714204250,2sleep2715636-21/usr/sbin/munin12:35:182
664045250,3sleep00-21swapper/011:48:340
584928250,1sleep10-21swapper/110:34:491
573092250,1sleep20-21swapper/210:20:222
488879251,1sleep20-21swapper/209:00:262
485880250,2sleep10-21swapper/109:00:181
477082250,1sleep00-21swapper/008:50:190
468865250,1sleep00-21swapper/008:45:130
442300250,3sleep00-21swapper/008:18:390
425459250,1sleep20-21swapper/208:00:222
675187240,1sleep20-21swapper/212:00:142
553250240,1sleep20-21swapper/210:05:032
543063240,1sleep00-21swapper/009:55:160
527402240,1sleep00-21swapper/009:40:160
416149240,1sleep20-21swapper/207:55:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional