You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 21:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Jan 11, 2026 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
484609212366,53sleep30-21swapper/307:06:353
67610621150,3sleep0221rcuc/010:05:270
48458928756,9sleep10-21swapper/107:06:201
47998028553,6sleep00-21swapper/007:05:030
48465428452,9sleep20-21swapper/207:07:102
485011992623,1cyclictest0-21swapper/311:25:013
8269902230,2sleep00-21swapper/012:35:170
485011992320,1cyclictest0-21swapper/310:15:133
485011992219,1cyclictest0-21swapper/309:15:123
485011992219,1cyclictest0-21swapper/307:36:583
48501199221,7cyclictest771878-21taskset11:40:123
48501199209,8cyclictest512551-21taskset07:30:233
485011992018,1cyclictest0-21swapper/309:45:183
485011992017,1cyclictest0-21swapper/309:05:173
48501199201,11cyclictest640133-21taskset09:35:113
48501199199,7cyclictest734356-21taskset11:03:053
48501199198,9cyclictest692273-21turbostat10:29:563
48501199195,10cyclictest750281-21turbostat11:20:003
485011991917,1cyclictest0-21swapper/312:10:243
485011991917,1cyclictest0-21swapper/308:55:123
485011991916,1cyclictest0-21swapper/308:20:113
485011991915,2cyclictest0-21swapper/308:45:233
485011991915,2cyclictest0-21swapper/308:45:223
485011991912,4cyclictest724002-21taskset10:54:563
48501199191,12cyclictest768040-21taskset11:35:163
48501199191,12cyclictest666066-21taskset10:00:013
48501199189,6cyclictest586960-21turbostat08:45:003
485011991815,1cyclictest0-21swapper/311:45:203
485011991815,1cyclictest0-21swapper/311:30:153
485011991815,1cyclictest0-21swapper/310:40:173
485011991815,1cyclictest0-21swapper/310:35:163
485011991815,1cyclictest0-21swapper/307:10:183
48501199181,14cyclictest810364-21taskset12:20:133
485011991811,4cyclictest576412-21taskset08:34:283
48501199181,13cyclictest0-21swapper/312:30:193
48501199181,13cyclictest0-21swapper/308:50:123
48501199181,11cyclictest539861-21taskset08:00:013
485011991715,1cyclictest0-21swapper/307:20:003
485011991714,2cyclictest0-21swapper/312:05:173
485011991714,1cyclictest0-21swapper/312:25:153
485011991714,1cyclictest0-21swapper/307:20:113
485011991713,2cyclictest0-21swapper/310:10:163
48501199171,12cyclictest0-21swapper/311:25:123
48501199171,12cyclictest0-21swapper/307:45:123
485011991710,3cyclictest0-21swapper/310:46:573
6933682161,10sleep2694221-21grep10:25:182
48501199167,4cyclictest5295272sleep307:50:113
48501199166,6cyclictest544908-21taskset08:03:013
48501199161,9cyclictest613349-21taskset09:10:073
485011991613,1cyclictest0-21swapper/310:30:143
48501199161,13cyclictest0-21swapper/311:55:153
48501199161,13cyclictest0-21swapper/311:55:143
48501199161,11cyclictest825035-21chrt12:35:123
48501199161,11cyclictest571244-21sleep08:29:593
48501199160,14cyclictest0-21swapper/308:35:143
48501199160,14cyclictest0-21swapper/308:08:533
48501199160,13cyclictest0-21swapper/312:00:213
48501199160,13cyclictest0-21swapper/311:50:193
48501199160,13cyclictest0-21swapper/311:11:463
48501199160,13cyclictest0-21swapper/309:00:203
8018032151,9sleep0801849-21ntpq12:05:200
7353572151,9sleep1736850-21sort11:05:171
6545922151,4sleep2654640-21ntpq09:45:222
5657682153,8sleep1248-21systemd-journal08:22:201
5282102151,10sleep0528261-21ntpq07:45:190
48501199151,9cyclictest808325-21taskset12:15:283
48501199151,9cyclictest739666-21chrt11:08:253
48501199151,9cyclictest639511-21taskset09:30:433
48501199151,9cyclictest554835-21taskset08:10:233
485011991512,2cyclictest0-21swapper/307:40:133
485011991512,2cyclictest0-21swapper/307:40:123
485011991512,1cyclictest0-21swapper/310:00:133
48501199151,10cyclictest729293-21sleep10:59:593
48501199151,10cyclictest629346-21chrt09:25:093
48501199151,10cyclictest625434-21chrt09:20:173
48501199151,10cyclictest508005-21chrt07:28:253
48501199150,12cyclictest0-21swapper/310:20:173
48501199150,12cyclictest0-21swapper/309:40:203
7876682141,3sleep1787856-21gltestperf11:55:151
7876682141,3sleep1787856-21gltestperf11:55:151
7657522140,1sleep10-21swapper/111:31:331
6717542141,9sleep2671876-21cut10:05:142
6616292141,8sleep1663161-21cut09:55:191
6246402141,9sleep2625433-21cut09:20:172
6100372141,8sleep2611841-21tr09:05:172
5404832142,8sleep1540631-21cat08:00:161
5038522141,8sleep2505492-21cut07:25:172
48501199149,2cyclictest0-21swapper/310:05:123
485011991411,2cyclictest0-21swapper/309:50:153
48501199140,12cyclictest0-21swapper/308:15:153
8036312131,7sleep1804266-21kthreadcore12:10:131
7660282131,8sleep2766032-21cron11:34:592
7098902131,8sleep2711605-21cut10:40:172
5229542131,8sleep1523008-21ntpq07:40:201
5229542131,8sleep1523008-21ntpq07:40:201
7995492121,7sleep1801303-21kthreadcore12:05:171
7861412121,5sleep0786188-21ntpq11:50:210
7123872121,5sleep0712500-21ntpq10:40:190
6555502120,1sleep10-21swapper/109:49:581
6120912122,6sleep1612139-21ntpq09:05:201
5809962121,7sleep1581131-21timerwakeupswit08:35:271
5508522121,7sleep1551041-21gltestperf08:10:141
5354212121,7sleep2536846-21kthreadcore07:55:162
5089682121,6sleep0510185-21kthreadcore07:30:170
8234672111,5sleep1823668-21grep12:30:171
7975362110,1sleep20-21swapper/212:04:222
7964972111,5sleep1796586-21ntpq12:00:201
7803472110,7sleep1780771-21needreboot11:45:171
7291112110,1sleep00-21swapper/010:58:140
6470952111,5sleep2648869-21cut09:40:172
5966752111,6sleep1596746-21find08:50:261
5406322114,3sleep2541304-21kthreadcore08:00:172
5347892111,4sleep1534808-21apt-get07:55:101
5248802111,6sleep1526398-21kthreadcore07:45:171
5183972111,5sleep0518419-31find07:36:580
8238032101,4sleep0823857-21ntpq12:30:210
8192112101,5sleep01-21systemd12:30:010
8143142101,4sleep2814529-21df_inode12:25:142
8128512101,4sleep1812964-21ntpq12:20:151
7613502101,5sleep0761547-21grep11:30:120
7606612100,4sleep2760654-21fwupd11:28:222
7492212101,4sleep2749272-21ntpq11:15:202
7455882101,5sleep1745796-21idleruntime11:15:151
7440892101,4sleep1744142-21ntp_states11:10:221
7347742101,3sleep0734793-21apt-get11:05:120
7072502101,4sleep2707297-21ntpq10:35:202
7033062101,4sleep1703325-21apt-get10:35:101
6922632101,4sleep1692287-21munin-run10:25:011
6662342101,3sleep00-21swapper/010:00:110
6610252101,4sleep0661044-21apt-get09:55:140
6298242101,4sleep2629893-21irqstats09:25:132
6225372103,3sleep0622584-21ntpq09:15:220
5976262101,4sleep2597645-21apt-get08:55:122
5702732101,4sleep2570330-21ntp_states08:25:212
5601502101,4sleep2560377-21users08:15:272
5505622101,3sleep2550581-21apt-get08:10:112
5335532101,5sleep0533600-21ntpq07:50:210
5195682101,5sleep2520747-21tr07:40:172
5195682101,5sleep2520747-21tr07:40:162
5184322101,4sleep1518419-31find07:36:581
4979462101,4sleep1498199-21df07:20:121
814806291,4sleep1815496-21grep12:25:171
760801291,4sleep0760821-21munin-run11:30:010
750547290,4sleep0750566-21apt-get11:20:100
734511291,4sleep1734629-21cat11:05:011
719302291,4sleep2719496-21expr10:50:152
713776291,4sleep0713795-21apt-get10:45:120
671209291,4sleep1671258-21/usr/sbin/munin10:05:001
668280291,4sleep2669989-21wc10:00:162
659778291,4sleep0659830-21ld09:50:200
649317291,3sleep1649366-21ntpq09:40:191
645367291,3sleep0645386-21apt-get09:40:100
640116290,3sleep20-21swapper/209:35:122
623518291,4sleep2623554-21perf09:20:002
608117291,3sleep0608136-21apt-get09:05:120
607857291,4sleep1607905-21munin-run09:05:011
573137291,4sleep1574918-21tr08:30:181
572050291,4sleep0573136-21grep08:30:170
519287291,4sleep0519496-21hddtemp_smartct07:40:150
519287291,4sleep0519496-21hddtemp_smartct07:40:150
510188291,4sleep1511934-21grep07:30:181
508686291,4sleep2508895-21hddtemp_smartct07:30:152
506971291,4sleep0507109-21netstat07:25:190
829979280,3sleep1830038-21sendmail-msp12:40:001
819719280,3sleep2819228-21apt-get12:30:022
808406283,3sleep0151ktimers/012:16:300
807174281,4sleep0553-21dbus-daemon12:10:200
807070281,2sleep2807118-21ntpq12:10:192
772158281,3sleep1772190-21kthreadcore11:40:151
766288280,3sleep1766307-21apt-get11:35:111
750569280,1sleep10-21swapper/111:20:111
729546281,3sleep2729558-21apt-get11:00:132
728112281,3sleep1728237-21ntpq10:55:191
718590280,1sleep10-21swapper/110:48:001
713848281,4sleep20-21swapper/210:45:132
712501281,2sleep1712547-21ntpq10:40:201
695975281,3sleep0696687-21missed_timers10:25:190
692801281,2sleep10-21swapper/110:25:131
686669280,1sleep10-21swapper/110:15:471
677495281,3sleep0679130-21cut10:10:170
638966281,3sleep2639081-21gzip09:30:232
597952281,3sleep10-21swapper/108:55:141
593317281,3sleep0594999-21cut08:50:170
581483280,1sleep20-21swapper/208:37:272
576446280,1sleep20-21swapper/208:34:562
566243281,2sleep00-21swapper/008:25:120
538902280,1sleep00-21swapper/007:55:210
503183280,3sleep1503201-21apt-get07:25:121
496955281,3sleep2497158-21unixbench-2d07:15:272
485185281,4sleep0485197-21apt-get07:10:130
813968271,2sleep00-21swapper/012:24:510
813644270,1sleep20-21swapper/212:20:442
781832270,1sleep20-21swapper/211:49:512
778562271,3sleep00-21swapper/011:45:160
771585270,2sleep0771604-21apt-get11:40:110
766310270,4sleep20-21swapper/211:35:122
756358271,2sleep10-21swapper/111:25:171
743862271,3sleep2743987-21/usr/sbin/munin11:10:192
723770270,1sleep10-21swapper/110:52:021
707570273,1sleep00-21swapper/010:35:250
701892271,2sleep1702029-21ntpq10:30:181
698633271,3sleep20-21swapper/210:30:152
687289271,2sleep00-21swapper/010:20:110
672673270,3sleep1674370-21grep10:05:171
655250270,1sleep00-21swapper/009:46:120
640110271,3sleep10-21swapper/109:35:111
617309270,4sleep20-21swapper/209:10:192
603440271,2sleep20-21swapper/209:00:172
591813270,1sleep00-21swapper/008:45:400
591813270,1sleep00-21swapper/008:45:390
587227271,2sleep10-21swapper/108:45:131
587227271,2sleep10-21swapper/108:45:121
586946271,3sleep1587160-21latency_hist08:45:011
585974271,2sleep2586022-21ntpq08:40:212
561593271,2sleep00-21swapper/008:20:150
556581272,2sleep00-21swapper/008:15:180
555497270,1sleep00-21swapper/008:14:180
545327270,2sleep00-21swapper/008:05:110
518246271,3sleep2518247-21pool-udisksd07:35:352
497924270,3sleep0497943-21apt-get07:20:110
496638270,2sleep00-21swapper/007:15:210
825013260,2sleep20-21swapper/212:35:122
808513260,1sleep20-21swapper/212:17:482
797309260,1sleep00-21swapper/012:01:260
792092260,1sleep00-21swapper/011:56:500
792092260,1sleep00-21swapper/011:56:500
787378260,2sleep20-21swapper/211:55:122
787378260,2sleep20-21swapper/211:55:122
786955260,1sleep20-21swapper/211:52:582
786770260,1sleep10-21swapper/111:50:381
775653261,2sleep20-21swapper/211:40:202
754587260,2sleep20-21swapper/211:20:212
744843260,1sleep00-21swapper/011:12:440
739645260,1sleep20-21swapper/211:08:092
724829261,2sleep20-21swapper/210:55:152
689408260,3sleep2691070-21cat10:20:162
686136261,2sleep2561-21systemd-logind10:15:242
683259261,2sleep00-21swapper/010:15:180
676743260,1sleep10-21swapper/110:10:061
665014260,3sleep20-21swapper/209:55:242
660428260,1sleep10-21swapper/109:50:451
660142260,2sleep20-21swapper/209:50:262
638869260,2sleep00-21swapper/009:30:200
630942261,2sleep00-21swapper/009:25:160
628743260,1sleep10-21swapper/109:20:561
627177261,2sleep00-21swapper/009:20:190
618138260,4sleep10-21swapper/109:12:381
606944260,3sleep00-21swapper/009:00:230
597905261,2sleep00-21swapper/008:55:130
597072260,4sleep20-21swapper/208:51:192
589077260,2sleep2421rcuc/208:45:182
589077260,2sleep2421rcuc/208:45:182
586753260,1sleep00-21swapper/008:42:340
580716260,2sleep00-21swapper/008:35:220
560055261,2sleep10-21swapper/108:15:261
550088260,1sleep10-21swapper/108:07:201
545017260,1sleep00-21swapper/008:04:240
534376260,4sleep10-21swapper/107:53:061
526401261,2sleep20-21swapper/207:45:182
490811260,2sleep1490193-21apt-get07:15:021
490031260,1sleep20-21swapper/207:13:112
486142261,2sleep10-21swapper/107:10:181
808298250,1sleep10-21swapper/112:15:071
801305250,1sleep20-21swapper/212:05:182
766579250,1sleep00-21swapper/011:35:140
749299250,1sleep00-21swapper/011:15:220
723002250,1sleep00-21swapper/010:50:210
637579250,1sleep10-21swapper/109:30:181
623274250,1sleep10-21swapper/109:16:581
570221250,1sleep10-21swapper/108:25:201
502726250,3sleep20-21swapper/207:22:332
729821240,1sleep00-21swapper/011:00:150
697813240,1sleep00-21swapper/010:30:010
677100240,1sleep20-21swapper/210:10:142
640103240,1sleep00-21swapper/009:35:060
629615240,1sleep10-21swapper/109:25:131
613638240,1sleep00-21swapper/009:10:130
561319240,1sleep20-21swapper/208:20:132
545111240,1sleep20-21swapper/208:05:012
529510240,1sleep20-21swapper/207:50:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional