You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Apr 07, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3264229076,9sleep20-21swapper/219:07:032
34428860,9sleep30-21swapper/319:09:133
3252728674,8sleep00-21swapper/019:05:330
34228163,14sleep10-21swapper/119:09:111
519992726,1cyclictest0-21swapper/100:25:581
51999270,2cyclictest0-21swapper/123:26:441
51599264,7cyclictest0-21swapper/022:55:150
51999241,1cyclictest0-21swapper/123:55:021
51599244,1cyclictest141rcu_preempt22:20:470
523992320,2cyclictest0-21swapper/221:28:162
52399220,10cyclictest0-21swapper/200:15:122
51999220,20cyclictest0-21swapper/122:30:141
515992221,1cyclictest0-21swapper/022:34:040
52399215,16cyclictest32-21ksoftirqd/200:14:552
52399215,15cyclictest17841-21/usr/sbin/munin23:00:252
51599214,2cyclictest225-21systemd-journal20:25:140
132562210,1sleep125-21ksoftirqd/119:35:141
523992019,0cyclictest0-21swapper/221:59:252
51599209,6cyclictest12475-21perf00:14:590
51599206,7cyclictest0-21swapper/020:40:120
52399184,13cyclictest1160-21kworker/u8:000:32:202
52399180,0cyclictest0-21swapper/222:40:132
51999181,1cyclictest25496-21diskmemload23:07:411
51999181,1cyclictest25496-21diskmemload23:07:401
51999180,14cyclictest0-21swapper/100:32:391
51599189,4cyclictest12676-21apt-get22:00:140
51599184,7cyclictest25091-21ssh23:25:000
51599180,0cyclictest0-21swapper/023:14:090
54272170,0chrt0-21swapper/122:15:101
52699173,8cyclictest5479-21idleruntime-cro19:20:013
526991716,1cyclictest0-21swapper/323:21:323
52699170,1cyclictest25496-21diskmemload22:43:043
523991716,1cyclictest0-21swapper/200:35:242
523991714,3cyclictest0-21swapper/221:50:232
523991711,5cyclictest13843-21memory22:00:192
52399170,15cyclictest0-21swapper/222:32:542
52399170,15cyclictest0-21swapper/221:48:392
51999173,8cyclictest0-21swapper/119:25:131
519991715,2cyclictest0-21swapper/100:18:221
519991715,1cyclictest0-21swapper/121:55:181
51999170,1cyclictest0-21swapper/121:54:231
51999170,17cyclictest0-21swapper/121:45:171
51999170,16cyclictest0-21swapper/123:53:211
51599179,2cyclictest5564-21wc21:55:210
515991713,4cyclictest13-21ksoftirqd/023:38:580
515991711,5cyclictest13335-21run-parts22:19:590
52399162,6cyclictest0-21swapper/223:50:142
519991611,5cyclictest38150irq/131-snd_hda21:38:061
51999161,11cyclictest0-21swapper/119:15:001
51599168,2cyclictest22347-21awk22:05:210
51599167,5cyclictest22865-21idleruntime-cro22:44:590
51599167,3cyclictest22565-21tr22:25:140
515991613,2cyclictest29007-21cat21:30:250
515991610,1cyclictest12465-21grep21:40:170
51599160,15cyclictest0-21swapper/000:34:120
52699155,4cyclictest0-21swapper/321:20:153
526991511,2cyclictest31272-21apache221:25:063
52699150,13cyclictest0-21swapper/323:07:253
52699150,13cyclictest0-21swapper/323:07:243
52399151,14cyclictest0-21swapper/221:36:222
51999159,5cyclictest506-21cyclictest22:22:291
51999150,0cyclictest0-21swapper/122:27:201
51599158,2cyclictest28371-21cat23:45:180
51599157,2cyclictest24799-21wc21:05:260
515991514,1cyclictest0-21swapper/022:47:190
51599151,1cyclictest0-21swapper/021:48:490
515991510,2cyclictest31270-21apache223:44:500
52699143,11cyclictest1142-21runrttasks22:39:593
526991410,3cyclictest21090-21wc19:50:203
52699140,1cyclictest0-21swapper/323:11:203
52699140,1cyclictest0-21swapper/300:17:203
52699140,12cyclictest4351-21basename20:25:133
523991413,1cyclictest0-21swapper/223:10:242
52399141,12cyclictest0-21swapper/223:24:442
51999141,12cyclictest7774-21ssh22:55:131
519991411,2cyclictest25496-21diskmemload23:03:221
519991410,3cyclictest30310-21sed20:10:161
519991410,3cyclictest14645-21sed20:45:171
519991410,2cyclictest19657-21wc21:45:011
51999140,14cyclictest0-21swapper/100:36:051
51599148,2cyclictest0-21swapper/023:03:590
51599146,7cyclictest17097-21munin-node21:19:250
515991413,0cyclictest0-21swapper/022:51:090
515991410,3cyclictest12211-21grep19:30:260
515991410,3cyclictest11810-21tr23:55:140
51599140,14cyclictest0-21swapper/023:51:050
51599140,0cyclictest0-21swapper/023:26:420
52699139,3cyclictest29218-21sleep00:25:003
52699131,2cyclictest141rcu_preempt00:04:583
526991312,1cyclictest0-21swapper/323:42:273
526991311,2cyclictest0-21swapper/323:03:483
526991311,1cyclictest823-21wc22:50:233
526991310,3cyclictest39-21ksoftirqd/323:55:123
526991310,2cyclictest32635-21taskset23:48:073
526991310,2cyclictest23943-21cat19:55:243
52699130,12cyclictest0-21swapper/321:50:593
52699130,0cyclictest0-21swapper/321:48:283
52399132,2cyclictest225-21systemd-journal22:14:592
523991310,2cyclictest8420-21grep19:25:142
523991310,2cyclictest4141-21sed19:15:162
523991310,2cyclictest32721-21grep20:15:162
523991310,2cyclictest27963-21sed23:45:162
523991310,2cyclictest26614-21grep23:25:182
523991310,2cyclictest24937-21grep20:00:152
523991310,2cyclictest21112-21grep22:05:152
523991310,2cyclictest12926-21grep23:55:212
523991310,2cyclictest12689-21cat19:35:012
51999139,2cyclictest15316-21cut19:40:131
51999138,4cyclictest5308-21grep00:10:191
51999136,5cyclictest25101-21runrttasks23:43:481
519991313,0cyclictest0-21swapper/122:44:011
51999131,11cyclictest31968-21ssh23:47:361
519991310,2cyclictest3413-21kill00:09:351
519991310,2cyclictest22105-21grep21:00:221
519991310,2cyclictest20719-21cat22:04:591
51999130,11cyclictest0-21swapper/120:05:141
51599139,2cyclictest475-21in:imuxsock21:25:010
51599138,3cyclictest25303-21head23:05:180
51599138,3cyclictest25303-21head23:05:170
515991310,2cyclictest3314-21grep21:35:110
52699129,2cyclictest7375-21taskset21:18:043
52699129,2cyclictest30836-21ls22:10:223
52699129,2cyclictest27970-21cut21:30:163
52699129,2cyclictest27957-21grep20:05:163
52699129,2cyclictest26778-21grep21:10:173
52699129,2cyclictest15999-21grep19:40:163
52699127,4cyclictest5125-21grep21:35:223
52699120,0cyclictest0-21swapper/323:26:193
52399129,2cyclictest7950-21head20:30:212
52399129,2cyclictest5106-21sed20:25:162
52399129,2cyclictest32307-21dump-pmu-power23:10:012
52399129,2cyclictest32307-21dump-pmu-power23:10:002
52399129,2cyclictest32095-21grep22:50:152
52399129,2cyclictest2751-21grep20:20:172
52399129,2cyclictest27076-21cat21:10:192
52399129,2cyclictest22235-21grep21:00:242
52399129,2cyclictest21676-21grep19:50:262
52399129,2cyclictest19254-21grep20:55:152
52399129,2cyclictest12125-21cut20:40:162
52399129,2cyclictest10223-21sed21:20:012
52399126,4cyclictest14997-21sh19:40:002
52399120,4cyclictest225-21systemd-journal23:44:592
52399120,2cyclictest0-21swapper/200:05:132
51999129,2cyclictest788-21head20:15:211
51999129,2cyclictest38150irq/131-snd_hda21:05:191
51999129,2cyclictest3747-21ssh22:52:151
51999129,2cyclictest3648-21tr19:15:141
51999129,2cyclictest28725-21wc21:30:211
51999129,2cyclictest24849-21wc22:45:221
51999128,3cyclictest6516-21grep20:30:001
51999124,7cyclictest5034-21latency_hist22:15:001
519991211,1cyclictest25496-21diskmemload22:06:051
51999120,12cyclictest25496-21diskmemload00:02:371
51599129,2cyclictest9551-21sort23:15:170
51599129,2cyclictest9158-21cut20:35:120
51599129,2cyclictest32725-21grep20:15:160
51599129,2cyclictest30553-21sort22:10:200
51599129,2cyclictest28665-21grep21:50:140
51599129,2cyclictest27031-21ssh21:10:180
51599129,2cyclictest21904-21grep00:00:270
51599129,2cyclictest21693-21grep19:50:260
51599129,2cyclictest19339-21head20:55:150
51599129,2cyclictest13318-21ssh00:15:140
51599128,3cyclictest5767-21apt-get19:20:130
51599128,3cyclictest14987-21cron19:40:000
51599128,2cyclictest19953-21wc19:50:010
51599120,4cyclictest0-21swapper/000:35:560
51599120,11cyclictest25735-21uname20:00:180
52699119,1cyclictest13184-21grep20:40:243
52699119,1cyclictest11974-21grep21:40:163
52699118,2cyclictest8583-21sed22:55:163
52699118,2cyclictest7487-21cut20:30:153
52699118,2cyclictest6831-21grep00:30:193
52699118,2cyclictest3982-21grep23:50:163
52699118,2cyclictest31795-21ssh00:06:323
52699118,2cyclictest31302-21head20:10:263
52699118,2cyclictest30043-21ssh22:29:573
52699118,2cyclictest23687-21ssh22:45:143
52699118,2cyclictest19940-21rm00:38:303
52699118,2cyclictest17634-21ssh22:21:493
52699118,2cyclictest11688-21rm21:59:393
52699118,2cyclictest11458-21ssh23:15:553
52699118,2cyclictest11033-21sed23:35:183
52699118,2cyclictest10551-21grep19:30:133
52699117,3cyclictest19360-21grep20:55:163
52699116,2cyclictest23385-21apt-get21:05:143
52699111,5cyclictest15004-21turbostat19:40:013
52699110,10cyclictest5655-21ssh23:31:533
52699110,10cyclictest29435-21ssh00:25:003
52399119,1cyclictest0-21swapper/222:18:312
52399118,2cyclictest9737-21uniq20:35:162
52399118,2cyclictest7035-21expr22:35:142
52399118,2cyclictest28921-21taskset20:05:312
52399118,2cyclictest28511-21grep21:30:192
52399118,2cyclictest28300-21sh00:24:192
52399118,2cyclictest26123-21ssh22:26:532
52399118,2cyclictest24157-21grep21:05:172
52399118,2cyclictest2123-21grep23:30:142
52399118,2cyclictest20649-21grep00:00:172
52399118,2cyclictest19442-21sh22:23:132
52399118,2cyclictest17954-21sed20:50:262
52399118,2cyclictest15816-21ssh23:19:192
52399118,2cyclictest14970-21ssh21:41:292
52399118,2cyclictest11346-21ssh22:56:412
52399118,2cyclictest10363-21egrep23:35:152
52399117,3cyclictest24517-21munin-run19:59:592
52399117,3cyclictest15904-21grep19:40:162
52399110,10cyclictest4363-21ssh21:15:422
52399110,10cyclictest28225-21sh22:47:392
51999118,2cyclictest9099-21tr23:15:141
51999118,2cyclictest6545-21sed19:20:161
51999118,2cyclictest31104-21ssh21:12:521
51999118,2cyclictest28667-21ssh00:24:351
51999118,2cyclictest22812-21cat19:55:141
51999118,2cyclictest21638-21sed19:50:261
51999118,2cyclictest20141-21wc21:25:191
51999118,2cyclictest19797-21ssh23:20:541
51999118,2cyclictest1916-21ssh21:15:111
51999118,2cyclictest18067-21chrt20:51:361
51999118,2cyclictest12240-21ssh21:20:261
51999118,2cyclictest1201-21sed23:10:161
51999118,2cyclictest0-21swapper/123:30:121
51599118,2cyclictest7310-21cut20:30:150
51599118,2cyclictest4621-21grep19:15:210
51599118,2cyclictest4260-21grep20:25:000
51599118,2cyclictest2647-21ssh00:09:060
51599118,2cyclictest21222-21ssh00:20:070
51599118,2cyclictest20360-21head21:25:210
51599118,2cyclictest14141-21grep20:45:150
51599118,2cyclictest12389-21ssh22:38:240
51599114,2cyclictest2113-21cpuspeed_turbos23:30:140
51599110,1cyclictest0-21swapper/000:25:330
52699108,2cyclictest20936-21ssh22:05:093
52699108,2cyclictest10337-21ssh22:17:413
52699108,1cyclictest4594-21basename00:10:153
52699108,1cyclictest31959-21cat20:15:113
52699108,1cyclictest30377-21ssh22:30:043
52699108,1cyclictest2332-21cat20:20:153
52699108,1cyclictest21311-21cat21:00:163
52699108,1cyclictest16418-21grep20:50:143
52699108,1cyclictest13856-21taskset20:45:133
52699108,1cyclictest12550-21ssh22:00:073
52699107,2cyclictest8447-21cstates19:25:143
52699107,2cyclictest7127-21tr19:20:213
52699107,2cyclictest10102-21load20:35:183
52699100,9cyclictest17687-21hostname19:45:123
52399108,1cyclictest5718-21seq19:20:022
52399108,1cyclictest29446-21ssh00:25:012
52399108,1cyclictest1030-21tr19:10:122
52399107,2cyclictest19073-21cat19:45:242
52399107,2cyclictest13929-21grep20:45:142
51999108,1cyclictest6805-21basename20:30:131
51999108,1cyclictest108702sleep120:35:561
51999107,2cyclictest9531-21ssh22:36:091
51999107,2cyclictest18856-21expr20:55:131
51999107,2cyclictest11723-21cut19:30:211
51999107,2cyclictest10710-21sh23:35:161
51999104,3cyclictest25021-21cpuspeed_turbos20:00:141
51999102,4cyclictest225-21systemd-journal20:20:131
51599108,1cyclictest9646-21grep19:25:240
51599108,1cyclictest2773-21chrt19:13:460
51599107,2cyclictest17432-21cat19:45:010
51599107,2cyclictest16748-21sed20:50:150
51599105,3cyclictest27438-21tr20:05:150
51599100,9cyclictest244412sleep019:59:140
51599100,9cyclictest0-21swapper/020:11:130
5269997,2cyclictest2812-21taskset19:14:153
5239997,1cyclictest29850-21cat20:10:132
5199996,2cyclictest11742-21tr20:40:141
5199990,7cyclictest18854-21pmu-power19:45:201
5159996,2cyclictest22097-21tr21:00:220
5269987,1cyclictest24918-21hostname20:00:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional