You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri May 08, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2166029481,9sleep20-21swapper/219:05:552
2194429350,39sleep00-21swapper/019:09:350
2185029364,9sleep30-21swapper/319:08:213
2179028564,6sleep10-21swapper/119:07:351
141292600,0sleep20-21swapper/222:06:352
2207899290,1cyclictest33642chrt23:55:491
2208599260,23cyclictest0-21swapper/300:26:123
2207899260,25cyclictest0-21swapper/123:42:171
22085992518,6cyclictest0-21swapper/321:10:493
317292240,1sleep30-21swapper/323:15:153
2207899234,3cyclictest23239-21idleruntime-cro21:15:001
2207599230,1cyclictest0-21swapper/023:09:520
22085992222,0cyclictest0-21swapper/300:11:063
2208599211,19cyclictest0-21swapper/323:41:573
2208599210,1cyclictest0-21swapper/321:58:113
2208599210,18cyclictest0-21swapper/322:50:143
2208399210,17cyclictest0-21swapper/222:18:052
2207599210,20cyclictest0-21swapper/023:45:120
2208399200,0cyclictest0-21swapper/219:20:142
22078992019,1cyclictest38150irq/131-snd_hda21:46:411
2207899200,1cyclictest0-21swapper/100:37:461
2207599200,20cyclictest0-21swapper/022:10:050
22083991914,5cyclictest55650irq/127-enp2s0-21:50:142
2207599197,6cyclictest8940-21perf19:50:000
2207599195,8cyclictest10176-21basename22:05:000
22075991919,0cyclictest0-21swapper/000:27:570
22075991918,1cyclictest0-21swapper/022:20:440
2207599191,1cyclictest0-21swapper/000:31:260
22078991817,1cyclictest11981-21ssh00:00:571
22075991817,1cyclictest0-21swapper/021:51:470
2207599180,18cyclictest0-21swapper/022:32:110
2208599175,10cyclictest0-21swapper/322:40:123
22083991713,2cyclictest0-21swapper/222:40:552
2208399171,1cyclictest0-21swapper/223:54:252
2208399170,1cyclictest0-21swapper/200:10:152
2208399170,17cyclictest0-21swapper/223:45:422
2208399170,15cyclictest0-21swapper/223:30:292
2207599170,1cyclictest0-21swapper/022:48:190
2208599162,10cyclictest39-21ksoftirqd/323:05:143
2208599161,14cyclictest0-21swapper/300:39:573
22083991612,3cyclictest21991-21sh00:07:072
2208399160,16cyclictest0-21swapper/223:01:192
2208399160,16cyclictest0-21swapper/221:47:152
22078991613,2cyclictest18495-21head00:05:161
2207899160,3cyclictest0-21swapper/123:38:361
2207899160,15cyclictest0-21swapper/100:15:561
2207899160,13cyclictest0-21swapper/123:50:131
2207599165,6cyclictest2907-21apt20:45:000
2207599164,7cyclictest18800-21munin-run00:24:590
2208599159,5cyclictest19049-21munin-node23:03:083
2208599154,4cyclictest0-21swapper/300:15:133
2208599152,3cyclictest0-21swapper/322:15:463
22085991511,3cyclictest2363-21wc20:40:253
22083991514,1cyclictest0-21swapper/221:10:212
22083991512,3cyclictest32-21ksoftirqd/221:28:212
2208399150,15cyclictest0-21swapper/221:42:282
2207899152,10cyclictest38150irq/131-snd_hda21:00:131
22078991515,0cyclictest0-21swapper/122:40:261
22078991514,1cyclictest0-21swapper/123:12:331
2207899151,12cyclictest0-21swapper/123:05:141
22075991515,0cyclictest0-21swapper/021:58:090
2207599150,14cyclictest0-21swapper/020:05:120
2208599146,4cyclictest1142-21runrttasks20:25:183
2208599145,4cyclictest225-21systemd-journal21:52:063
22085991410,3cyclictest4662-21tr19:40:143
2208399142,5cyclictest28210-21latency_hist22:35:002
2208399141,10cyclictest537-21snmpd21:57:382
22083991410,3cyclictest4320-21grep00:35:172
22083991410,3cyclictest32750-21grep19:30:172
22083991410,3cyclictest1987-21ls20:40:202
2208399140,12cyclictest0-21swapper/200:21:032
22075991414,0cyclictest0-21swapper/023:10:360
22075991413,1cyclictest0-21swapper/023:33:400
22075991410,3cyclictest10092-21ssh21:45:180
2208599139,3cyclictest3492-21apt00:35:003
2208599139,2cyclictest5774-21grep20:50:143
2208599139,2cyclictest20585-21ssh00:06:073
2208599134,8cyclictest29202-21df_inode22:35:163
22085991310,2cyclictest9415-21grep21:25:213
22085991310,2cyclictest544-21head21:20:173
22085991310,2cyclictest4277-21uname20:45:203
22085991310,2cyclictest30568-21awk20:35:003
22085991310,2cyclictest24447-21grep20:20:163
22085991310,2cyclictest24152-21grep23:30:163
22085991310,2cyclictest23362-21sh23:30:003
22085991310,2cyclictest14978-21dump-pmu-power23:25:003
2208399139,3cyclictest20566-21wc22:10:242
2208399139,3cyclictest17114-21grep20:05:162
2208399138,4cyclictest30380-21ssh22:35:272
22083991310,2cyclictest9572-21ssh23:40:222
22083991310,2cyclictest4427-21tail00:15:262
22083991310,2cyclictest32511-21grep23:35:152
22083991310,2cyclictest27291-21cat00:30:002
22083991310,2cyclictest20962-21rm22:50:002
22083991310,2cyclictest14961-21cron23:25:002
22083991310,2cyclictest13798-21grep20:00:012
22083991310,2cyclictest13774-21cat21:05:202
2207899139,3cyclictest19156-21ssh21:50:221
2207899139,3cyclictest11772-21cat22:05:191
2207899132,9cyclictest25-21ksoftirqd/120:40:121
22078991312,1cyclictest0-21swapper/121:26:251
22078991312,1cyclictest0-21swapper/100:13:471
2207899131,1cyclictest14851-21diskmemload22:54:471
2207899131,1cyclictest0-21swapper/121:15:121
22078991311,2cyclictest0-21swapper/122:49:581
22078991310,2cyclictest8484-21sed23:20:221
22078991310,2cyclictest7855-21sed23:00:261
22078991310,2cyclictest3792-21grep20:45:161
22078991310,2cyclictest23230-21grep19:10:191
22078991310,2cyclictest18466-21sed21:30:281
2207899130,1cyclictest0-21swapper/123:46:171
2207599139,3cyclictest29888-21grep19:25:150
2207599139,3cyclictest1904-21cat19:35:000
22075991310,2cyclictest7564-21ls23:40:000
22075991310,2cyclictest26375-21wc19:15:280
22075991310,2cyclictest24245-21df20:20:140
22075991310,2cyclictest24132-21sed19:10:280
22075991310,2cyclictest23401-21cron20:20:010
22075991310,2cyclictest15991-21cron23:45:000
22075991310,2cyclictest14381-21grep21:05:270
22075991310,2cyclictest10797-21grep19:50:250
2207599130,12cyclictest20822-21apt20:15:000
2207599130,12cyclictest0-21swapper/021:15:270
57672120,0chrt0-21swapper/000:16:240
299342120,1sleep00-21swapper/022:55:130
2208599129,2cyclictest26993-21cron19:20:003
2208599129,2cyclictest24074-21grep23:10:223
2208599129,2cyclictest17434-21cut21:30:183
2208599129,2cyclictest16081-21ls20:05:003
2208599129,2cyclictest13450-21grep21:05:173
2208599129,2cyclictest11770-21expr00:20:203
2208599129,2cyclictest11489-21sed22:05:173
2208599129,2cyclictest10589-21grep21:00:143
2208599128,3cyclictest13036-21tr22:25:183
2208599128,2cyclictest19326-21cut20:10:153
2208599122,8cyclictest11288-21latency_hist22:25:003
2208599120,10cyclictest0-21swapper/322:10:553
2208399129,2cyclictest5027-21cut19:40:162
2208399129,2cyclictest19806-21grep20:10:192
2208399129,2cyclictest12485-21dump-pmu-power21:05:002
2208399129,2cyclictest1179-21grep23:55:162
2208399128,3cyclictest32245-21ls23:15:202
2208399128,3cyclictest32035-21taskset21:20:082
2208399123,3cyclictest225-21systemd-journal21:34:062
2208399120,11cyclictest29660-21/usr/sbin/munin19:25:132
2207899129,2cyclictest8242-21tr20:55:141
2207899129,2cyclictest32698-21sh21:40:001
2207899129,2cyclictest25611-21awk23:30:271
2207899129,2cyclictest19792-21cut22:10:181
2207899129,2cyclictest10163-21basename22:05:001
2207899126,6cyclictest18197-21kworker/1:200:25:291
2207899120,12cyclictest0-21swapper/123:15:191
2207599129,2cyclictest7652-21sh20:55:000
2207599129,2cyclictest27180-21grep00:10:170
2207599129,2cyclictest26781-21grep20:25:160
2207599129,2cyclictest1727-21sed23:55:190
2207599128,3cyclictest7328-21grep23:20:150
2207599128,2cyclictest3307-21grep20:45:140
2207599126,2cyclictest23234-21awk21:15:000
2208599118,2cyclictest6787-21grep19:45:153
2208599118,2cyclictest32192-21ssh23:35:123
2208599118,2cyclictest3021-21grep22:00:173
2208599118,2cyclictest29971-21ssh23:52:513
2208599118,2cyclictest29666-21ssh21:37:393
2208599118,2cyclictest29646-21ssh22:55:013
2208599118,2cyclictest29109-21ssh21:18:033
2208599118,2cyclictest2720-21grep19:35:163
2208599118,2cyclictest23384-21ssh22:31:213
2208599118,2cyclictest22391-21grep20:15:213
2208599118,2cyclictest17782-21ssh22:47:383
2208599118,2cyclictest1690-21sed21:40:183
2208599118,2cyclictest12769-21ssh00:01:333
2208599118,2cyclictest1222-21grep19:30:263
2208599117,3cyclictest23641-21ntpq19:10:223
2208599115,2cyclictest225-21systemd-journal20:05:073
22085991110,1cyclictest0-21swapper/323:45:133
2208599110,10cyclictest3229-21ssh23:55:443
2208599110,10cyclictest15780-21ssh21:48:593
2208399118,2cyclictest9032-21wc20:55:192
2208399118,2cyclictest6267-21ssh22:21:092
2208399118,2cyclictest3601-21grep19:35:252
2208399118,2cyclictest29765-21ssh22:55:072
2208399118,2cyclictest27573-21ssh00:30:052
2208399118,2cyclictest26695-21ssh22:52:542
2208399118,2cyclictest26594-21wc21:35:262
2208399118,2cyclictest25173-21cut19:15:172
2208399118,2cyclictest24460-21head20:20:162
2208399118,2cyclictest24152-21expr21:15:142
2208399118,2cyclictest16911-21ssh23:25:232
2208399118,2cyclictest16338-21ssh23:05:282
2208399118,2cyclictest12629-21sed22:25:152
2208399112,5cyclictest225-21systemd-journal20:20:012
2208399110,5cyclictest9984-21ssh00:00:172
2208399110,10cyclictest9600-21cat19:50:142
2207899119,1cyclictest26562-21grep19:15:281
2207899118,2cyclictest4924-21sh22:20:221
2207899118,2cyclictest32219-21egrep19:30:141
2207899118,2cyclictest31989-21rm21:20:061
2207899118,2cyclictest31794-21cat19:30:001
2207899118,2cyclictest30222-21ssh22:16:051
2207899118,2cyclictest29077-21sh21:56:071
2207899118,2cyclictest28366-21cat20:30:001
2207899118,2cyclictest2537-21sh00:34:171
2207899118,2cyclictest21947-21wc22:30:281
2207899118,2cyclictest17686-21which20:05:211
2207899118,2cyclictest15520-21ssh00:22:281
2207899116,2cyclictest6226-21ssh21:43:061
2207899113,4cyclictest225-21systemd-journal20:25:001
2207899110,9cyclictest0-21swapper/122:25:111
2207899110,2cyclictest0-21swapper/120:35:121
2207899110,10cyclictest32345-21chrt22:55:461
2207899110,10cyclictest28993-21expr22:35:161
2207899110,10cyclictest15423-21basename23:25:111
2207599118,2cyclictest9888-21sed00:00:170
2207599118,2cyclictest5916-21sh23:00:130
2207599118,2cyclictest5644-21ssh21:42:400
2207599118,2cyclictest5336-21grep19:40:190
2207599118,2cyclictest5191-21cut22:40:160
2207599118,2cyclictest32653-21expr21:20:150
2207599118,2cyclictest32016-21chrt20:35:200
2207599118,2cyclictest30206-21ssh21:38:040
2207599118,2cyclictest30155-21grep20:30:260
2207599118,2cyclictest29847-21ntpq22:35:210
2207599118,2cyclictest29798-21ssh23:52:430
2207599118,2cyclictest23443-21grep22:50:270
2207599118,2cyclictest2334-21ssh22:19:370
2207599118,2cyclictest16853-21grep21:30:160
2207599118,2cyclictest16080-21grep23:25:150
2207599118,2cyclictest1439-21sh23:16:020
2207599118,2cyclictest14024-21cut20:00:120
2207599118,2cyclictest13695-21ssh22:06:150
2207599118,2cyclictest0-21swapper/019:59:440
2207599117,2cyclictest9422-21cat21:25:210
2207599117,2cyclictest4577-21ssh00:35:180
2207599116,3cyclictest21976-21ssh00:07:060
2208599108,1cyclictest3872sleep320:39:083
2208599108,1cyclictest29132-21taskset19:21:483
2208599108,1cyclictest12955-21chrt19:55:253
2208599108,1cyclictest11418-21date19:55:003
2208599107,2cyclictest29665-21apt-get19:25:143
2208599100,9cyclictest98652sleep320:57:233
2208399108,1cyclictest28685-21expr20:30:142
2208399108,1cyclictest26440-21cat20:25:142
2208399108,1cyclictest22479-21ssh23:10:052
2208399108,1cyclictest1972-21ssh22:00:052
2208399108,1cyclictest14344-21expr20:00:132
2208399107,2cyclictest3309-21tr20:45:142
2208399107,2cyclictest31742-21kernelversion20:35:182
2208399107,2cyclictest22473-21chrt19:10:082
2208399104,1cyclictest6757-21apt-get19:45:142
2207899108,1cyclictest7421-21taskset20:52:141
2207899108,1cyclictest6219-21taskset19:41:431
2207899108,1cyclictest231602sleep120:16:581
2207899108,1cyclictest205982sleep120:12:221
2207899108,1cyclictest13069-21expr21:05:151
2207899108,1cyclictest11946-21expr19:55:151
2207899108,1cyclictest0-21swapper/120:00:121
2207899107,2cyclictest30056-21grep20:30:261
2207899107,2cyclictest2970-21wc19:35:181
2207899107,2cyclictest27624-21expr19:20:151
2207599108,1cyclictest1990-21dpkg19:35:120
2207599108,1cyclictest14748-21sh22:26:100
2207599107,2cyclictest8930-21wc20:55:180
2207599107,2cyclictest27964-21sort19:20:170
2207599107,2cyclictest10471-21cut21:00:130
220839997,1cyclictest5688-21egrep20:50:142
220789997,1cyclictest67102sleep119:45:051
220789997,1cyclictest111462sleep119:53:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional