You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 10:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Apr 28, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3018421200,0sleep20-21swapper/221:54:042
2789828762,9sleep30-21swapper/319:08:373
2784728674,8sleep00-21swapper/019:07:580
2769328560,9sleep20-21swapper/219:05:592
2786327962,13sleep10-21swapper/119:08:111
28101992720,5cyclictest21966-21kworker/3:222:23:333
28091992726,1cyclictest0-21swapper/022:55:460
2810199250,1cyclictest0-21swapper/300:15:553
2809399250,24cyclictest0-21swapper/123:57:401
81212240,0sleep20-21swapper/222:19:402
2810199240,23cyclictest21966-21kworker/3:223:08:433
2810199230,23cyclictest0-21swapper/322:56:213
2810199230,22cyclictest21966-21kworker/3:200:04:533
2809399230,0cyclictest0-21swapper/122:27:131
48012220,0sleep30-21swapper/321:57:553
28096992221,1cyclictest32-21ksoftirqd/223:17:362
285932210,2chrt311rcuc/222:50:262
2810199200,2cyclictest0-21swapper/300:08:073
2810199200,19cyclictest0-21swapper/321:15:163
2809699203,4cyclictest0-21swapper/221:36:272
2809699200,19cyclictest0-21swapper/222:05:582
2809399207,6cyclictest7235-21munin-node00:25:581
28093992015,4cyclictest38150irq/131-snd_hda00:05:331
2809399200,2cyclictest0-21swapper/121:27:541
28101991913,5cyclictest39-21ksoftirqd/323:46:373
2810199190,0cyclictest32220-21ntpq21:35:213
2809699190,19cyclictest0-21swapper/221:20:242
2809699190,16cyclictest0-21swapper/200:05:062
2809399190,2cyclictest0-21swapper/122:55:281
2809399190,1cyclictest0-21swapper/122:23:081
2809399190,19cyclictest0-21swapper/123:22:281
2810199182,3cyclictest13071-21kworker/u8:321:44:043
2809699188,3cyclictest0-21swapper/221:40:182
28096991817,1cyclictest0-21swapper/219:14:092
28093991817,1cyclictest0-21swapper/121:41:551
2809399180,18cyclictest0-21swapper/121:58:101
2809399180,18cyclictest0-21swapper/100:18:511
2809199180,17cyclictest0-21swapper/023:27:280
2809699173,6cyclictest21147-21diskmemload23:00:002
28093991717,0cyclictest0-21swapper/121:19:341
2809399171,13cyclictest0-21swapper/123:45:151
2809399170,16cyclictest0-21swapper/121:50:081
2809199177,5cyclictest0-21swapper/020:05:010
28101991616,0cyclictest0-21swapper/323:56:593
28101991614,2cyclictest39-21ksoftirqd/323:13:223
28101991614,2cyclictest0-21swapper/322:17:123
2810199160,2cyclictest0-21swapper/321:34:243
28096991614,1cyclictest16934-21munin-run22:25:002
28093991612,3cyclictest21147-21diskmemload00:36:271
2809399160,2cyclictest0-21swapper/122:51:401
2809399160,1cyclictest0-21swapper/100:04:341
2809199164,10cyclictest26827-31find00:07:540
28091991614,1cyclictest0-21swapper/023:33:420
28091991612,2cyclictest0-21swapper/023:37:580
2810199157,7cyclictest487-21avahi-daemon21:00:193
2810199150,13cyclictest0-21swapper/320:05:143
2809699153,1cyclictest0-21swapper/200:30:062
28093991510,1cyclictest21760-21wc23:05:271
2809399150,15cyclictest0-21swapper/123:17:471
2809399150,0cyclictest0-21swapper/100:13:151
28091991512,2cyclictest13-21ksoftirqd/022:53:070
2809199150,15cyclictest0-21swapper/023:15:420
68042140,0chrt0-21swapper/121:20:171
28101991412,1cyclictest26799-21kworker/3:100:10:543
28101991411,2cyclictest21125-21grep20:00:203
2809699149,4cyclictest20779-21taskset00:22:312
28096991413,1cyclictest0-21swapper/222:25:232
28096991411,2cyclictest14755-21sh23:20:332
2809699140,3cyclictest0-21swapper/221:47:102
2809699140,1cyclictest0-21swapper/222:11:402
2809399144,4cyclictest452-21systemd-logind19:35:221
28093991411,2cyclictest5750-21cut19:30:131
28093991410,3cyclictest11274-21wc22:40:201
28093991410,2cyclictest24912-21grep20:10:151
2809399140,14cyclictest0-21swapper/121:10:151
2809399140,13cyclictest28494-21basename23:10:141
2809199149,3cyclictest5702-21tr20:35:210
2809199149,3cyclictest2218-21wc19:20:220
2809199148,4cyclictest31566-21apt-config21:55:000
2809199148,4cyclictest21147-21diskmemload00:20:230
2809199146,7cyclictest27427-21sensors_fan22:30:240
2809199141,7cyclictest8974-21idleruntime-cro20:45:010
28091991413,1cyclictest13-21ksoftirqd/023:48:450
28091991410,2cyclictest13877-21tail19:45:200
2809199140,13cyclictest32215-21wc23:50:260
97032130,0sleep00-21swapper/021:21:540
2810199139,3cyclictest5130-21sed23:15:183
2810199139,3cyclictest14014-21grep23:20:223
28101991310,2cyclictest25524-21cat22:30:003
28101991310,2cyclictest23909-21tr21:50:153
28101991310,2cyclictest10212-21sed20:45:183
2810199130,1cyclictest0-21swapper/322:00:083
2809699133,5cyclictest225-21systemd-journal00:17:332
28096991310,2cyclictest32431-21grep20:25:162
28096991310,2cyclictest30095-21grep23:30:152
28096991310,2cyclictest20255-21grep20:00:152
2809699130,1cyclictest0-21swapper/221:31:092
2809699130,1cyclictest0-21swapper/200:14:562
2809399139,3cyclictest6572-21cron20:40:001
2809399139,3cyclictest13959-21tr23:40:131
2809399138,2cyclictest21147-21diskmemload23:32:591
28093991310,2cyclictest31307-21wc20:20:251
28093991310,2cyclictest30714-21chrt19:15:151
28093991310,2cyclictest15479-21tr21:45:171
2809199135,7cyclictest16909-21ssh22:25:000
28091991310,2cyclictest32567-21ssh00:10:170
28091991310,2cyclictest23209-21wc21:50:000
28091991310,2cyclictest18650-21wc19:55:190
28091991310,2cyclictest14508-21tr21:25:130
28091991310,2cyclictest14315-21uniq23:40:150
28091991310,2cyclictest13303-21wc20:50:240
28091991310,2cyclictest13260-21grep23:00:260
28091991310,2cyclictest12535-21awk19:45:000
2810199129,2cyclictest9712-21sed19:35:263
2810199129,2cyclictest9310-21grep00:35:143
2810199129,2cyclictest7048-21sort23:35:263
2810199129,2cyclictest6154-21cut21:20:143
2810199129,2cyclictest31437-21wc23:50:183
2810199129,2cyclictest26561-21grep22:30:173
2810199129,2cyclictest2411-21grep22:35:143
2810199129,2cyclictest22896-21grep23:25:263
2810199129,2cyclictest21923-21ssh21:29:543
2810199129,2cyclictest18388-21head22:05:253
2810199129,2cyclictest1806-21taskset22:54:393
2810199129,2cyclictest16330-21basename21:00:003
2810199129,2cyclictest15411-21grep19:50:143
2810199129,2cyclictest14888-21cron19:50:003
2810199128,3cyclictest12591-21sed20:50:163
2810199122,7cyclictest11088-21fschecks_time19:40:173
28101991211,1cyclictest21966-21kworker/3:222:14:143
2809699129,2cyclictest8694-21grep19:35:162
2809699129,2cyclictest4943-21grep19:25:252
2809699129,2cyclictest30239-21grep21:15:162
2809699129,2cyclictest3007-21ls19:24:592
2809699129,2cyclictest28987-21grep20:15:272
2809699129,2cyclictest26328-21expr22:30:152
2809699129,2cyclictest22070-21perf21:30:002
2809699129,2cyclictest20082-21cat21:05:202
2809699123,3cyclictest225-21systemd-journal20:48:332
2809699123,3cyclictest225-21systemd-journal19:55:002
2809699122,6cyclictest21147-21diskmemload23:45:162
2809699121,7cyclictest25263-21df20:10:172
2809699121,1cyclictest0-21swapper/221:13:322
28096991210,1cyclictest20033-21ssh23:05:142
28096991210,1cyclictest13491-21grep20:50:252
28096991210,1cyclictest0-21swapper/219:55:122
2809699120,3cyclictest0-21swapper/200:02:282
2809699120,11cyclictest5543-21grep23:35:152
2809399129,2cyclictest8867-21tail22:00:181
2809399129,2cyclictest6104-21grep23:35:181
2809399129,2cyclictest28501-21tr20:15:221
2809399129,2cyclictest26119-21tr22:30:151
2809399129,2cyclictest2331-21grep20:30:141
2809399129,2cyclictest19282-21grep19:55:261
2809399129,2cyclictest1654-21cut19:20:171
2809399129,2cyclictest16497-21tr00:20:151
2809399129,2cyclictest13953-21basename20:55:001
2809399128,3cyclictest22679-21ssh22:47:081
2809399128,3cyclictest16795-21grep19:50:261
2809399127,3cyclictest4024-21cut19:25:161
2809399126,2cyclictest22565-21ssh21:30:121
2809399125,6cyclictest18331-21apache223:54:361
2809399124,7cyclictest28921-21apache200:31:291
2809399120,12cyclictest21147-21diskmemload22:35:101
2809199129,2cyclictest7283-21tail19:30:250
2809199129,2cyclictest388-21tail20:25:170
2809199129,2cyclictest31377-21grep20:20:250
2809199129,2cyclictest2867-21sed20:30:160
2809199129,2cyclictest28469-21grep20:15:210
2809199129,2cyclictest25483-21tail20:10:170
2809199129,2cyclictest20836-21tail23:05:170
2809199128,3cyclictest14492-21grep23:20:270
2809199125,3cyclictest5353-21sendmail21:20:000
28091991210,1cyclictest61002sleep021:58:550
28091991210,1cyclictest294552sleep022:12:330
2809199120,0cyclictest0-21swapper/022:44:500
2810199119,1cyclictest6189-21sed20:35:273
2810199118,2cyclictest6819-21chrt19:30:213
2810199118,2cyclictest32108-21seq20:25:103
2810199118,2cyclictest29399-21cut23:30:133
2810199118,2cyclictest29130-21seq20:15:413
2810199118,2cyclictest28574-21grep19:10:143
2810199118,2cyclictest27899-21ssh21:13:493
2810199118,2cyclictest2771-21ssh00:30:293
2810199118,2cyclictest26713-21sed00:25:273
2810199118,2cyclictest19265-21sed19:55:263
2810199118,2cyclictest18973-21expr22:45:173
2810199118,2cyclictest18601-21ssh00:20:473
2810199118,2cyclictest17607-21ssh21:45:473
2810199118,2cyclictest15797-21ssh23:40:323
2810199118,2cyclictest13268-21sed23:00:263
2810199118,2cyclictest10353-21sort22:40:143
2810199110,10cyclictest5067-21taskset19:26:063
2809699119,1cyclictest9013-21sed20:45:002
2809699119,1cyclictest14485-21sed19:45:282
2809699118,2cyclictest5983-21expr19:30:152
2809699118,2cyclictest31571-21cat23:50:192
2809699118,2cyclictest26049-21rm23:27:492
2809699118,2cyclictest24797-21ssh00:25:152
2809699118,2cyclictest20992-21taskset22:45:482
2809699118,2cyclictest15250-21ssh22:04:322
2809699118,2cyclictest12213-21tail23:00:172
2809699112,2cyclictest141rcu_preempt00:39:532
2809699110,1cyclictest0-21swapper/223:59:362
2809399118,2cyclictest9136-21cat20:45:001
2809399118,2cyclictest722-21ssh22:15:131
2809399118,2cyclictest31993-21ssh22:14:321
2809399118,2cyclictest15831-21grep20:55:251
2809399118,2cyclictest13016-21grep19:45:161
2809399118,2cyclictest12609-21ssh23:00:201
2809399118,2cyclictest10687-21tr19:40:151
2809399117,3cyclictest22196-21wc23:25:201
2809199118,2cyclictest6494-21tr23:55:130
2809199118,2cyclictest3949-21cut19:25:160
2809199118,2cyclictest32123-21grep19:15:270
2809199118,2cyclictest30276-21tail23:10:260
2809199118,2cyclictest26503-21ssh21:12:420
2809199118,2cyclictest2152-21cut22:15:220
2809199118,2cyclictest21215-21ssh00:04:020
2809199118,2cyclictest20210-21ssh22:45:270
2809199118,2cyclictest19584-21ssh22:06:210
2809199118,2cyclictest16847-21tr21:00:120
2809199118,2cyclictest16802sleep022:35:090
2809199118,2cyclictest15229-21grep19:50:130
2809199118,2cyclictest10925-21grep00:35:260
2809199118,2cyclictest10776-21ssh22:01:020
2809199117,3cyclictest9717-21tr20:45:150
2809199117,3cyclictest15300-21head20:55:190
2809199114,4cyclictest22527-21/usr/sbin/munin21:30:160
2809199111,4cyclictest0-21swapper/000:25:150
2809199110,1cyclictest0-21swapper/021:45:000
2810199108,1cyclictest843-21dpkg19:20:023
2810199107,2cyclictest8652-21taskset20:41:013
2810199107,2cyclictest2386-21tr20:30:143
2810199107,2cyclictest21153-21awk21:09:593
2810199105,2cyclictest225-21systemd-journal19:19:593
2810199100,10cyclictest25246-21expr20:10:163
2809699108,2cyclictest31539-21taskset20:20:492
2809699108,1cyclictest28192-21ssh23:10:062
2809699108,1cyclictest230242sleep220:05:172
2809699108,1cyclictest17907-21find21:00:192
2809699108,1cyclictest13894-21cut23:40:132
2809699107,2cyclictest9912-21ssh22:40:062
2809699107,2cyclictest6122-21sed20:35:252
2809699107,2cyclictest31994-21ssh21:55:092
2809699107,2cyclictest1514-21chrt22:35:032
2809699107,2cyclictest11763-21cut19:40:232
2809699102,4cyclictest225-21systemd-journal19:20:002
2809399108,1cyclictest29477-21ntpq19:10:211
2809399108,1cyclictest189932sleep121:05:111
2809399107,2cyclictest2425-21ssh21:36:581
2809399107,2cyclictest17988-21ssh22:05:241
2809399101,8cyclictest0-21swapper/120:45:141
2809399100,9cyclictest32156-21cut20:25:151
2809199108,1cyclictest9827-21chrt19:36:180
2809199108,1cyclictest8535-21fschecks_time00:15:190
2809199108,1cyclictest560-21ssh00:30:080
2809199108,1cyclictest31153-21tr21:35:140
2809199107,2cyclictest29861-21grep19:10:260
2809199107,2cyclictest23541-21tr20:05:210
2809199107,2cyclictest18403-21ssh22:25:170
2809199103,3cyclictest225-21systemd-journal21:05:330
281019997,1cyclictest30466-21gltestperf20:20:163
280969997,1cyclictest39162sleep220:31:252
280969997,1cyclictest14987-21sh20:55:162
280939997,1cyclictest223622sleep120:05:041
280939997,1cyclictest20481-21expr20:00:161
280939997,1cyclictest17225-21cut21:00:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional