You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 17:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed May 20, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2483321130,0sleep10-21swapper/109:20:131
13955210086,9sleep00-21swapper/007:08:590
1401429279,9sleep20-21swapper/207:09:442
1345828563,6sleep30-21swapper/307:05:273
1373228264,14sleep10-21swapper/107:06:061
1412899290,3cyclictest141rcu_preempt10:11:193
1412899290,3cyclictest141rcu_preempt10:11:193
1412399240,1cyclictest0-21swapper/112:30:131
1412899230,22cyclictest0-21swapper/311:07:573
1412399230,2cyclictest0-21swapper/110:00:121
1412899220,3cyclictest0-21swapper/310:32:163
14125992220,2cyclictest0-21swapper/209:59:132
1412399220,1cyclictest0-21swapper/111:06:271
1411999220,19cyclictest0-21swapper/010:32:170
1412599210,15cyclictest29669-21df10:40:142
1412399215,3cyclictest141rcu_preempt09:30:011
14123992117,4cyclictest241rcuc/107:25:251
1411999210,20cyclictest0-21swapper/009:36:470
14125992019,1cyclictest0-21swapper/209:50:152
1412599200,14cyclictest0-21swapper/209:43:252
1412399202,1cyclictest23244-21ssh09:57:421
1412399200,1cyclictest0-21swapper/111:50:021
1412399200,1cyclictest0-21swapper/110:38:501
1411999200,19cyclictest0-21swapper/009:32:380
1412399191,9cyclictest0-21swapper/111:10:141
14123991918,1cyclictest7076-21diskmemload09:44:011
1412399190,19cyclictest0-21swapper/109:30:531
1412899183,10cyclictest39-21ksoftirqd/309:59:463
1412899180,18cyclictest0-21swapper/311:10:053
1412899180,0cyclictest0-21swapper/310:55:473
1412599182,15cyclictest12540-21cstates10:30:132
14123991812,6cyclictest0-21swapper/111:03:511
1412399180,1cyclictest0-21swapper/111:27:101
14128991712,5cyclictest10377-21apache212:17:543
1412599170,17cyclictest0-21swapper/209:14:302
1412599170,16cyclictest0-21swapper/211:17:302
14123991715,2cyclictest7076-21diskmemload12:06:331
1412399171,14cyclictest7076-21diskmemload12:29:001
1412399170,2cyclictest0-21swapper/111:21:151
1412399170,17cyclictest0-21swapper/111:47:411
14119991715,1cyclictest13-21ksoftirqd/010:05:230
14119991710,4cyclictest21723-21apt-get07:25:130
1412899162,9cyclictest7076-21diskmemload11:04:093
1412899161,9cyclictest0-21swapper/307:25:143
1412899161,5cyclictest0-21swapper/311:24:283
1412899160,16cyclictest0-21swapper/310:40:363
1412599160,13cyclictest0-21swapper/211:12:472
1412399165,6cyclictest21907-21apt-get10:55:141
14123991616,0cyclictest0-21swapper/110:15:111
14123991615,1cyclictest7076-21diskmemload10:44:241
1412399160,15cyclictest0-21swapper/111:35:111
1411999165,1cyclictest141rcu_preempt09:56:450
14119991615,1cyclictest0-21swapper/012:10:270
14119991614,2cyclictest13-21ksoftirqd/011:17:530
1411999160,1cyclictest0-21swapper/011:45:110
1411999160,16cyclictest0-21swapper/008:16:050
1412899153,11cyclictest7076-21diskmemload10:18:503
1412899153,0cyclictest0-21swapper/311:34:453
14128991515,0cyclictest0-21swapper/309:15:123
1412899151,13cyclictest6711-21sensors10:45:243
14128991511,3cyclictest39-21ksoftirqd/311:17:093
1412899150,1cyclictest141rcu_preempt10:05:213
1412599153,11cyclictest1142-21runrttasks10:15:512
14125991514,1cyclictest0-21swapper/211:46:142
14125991510,5cyclictest0-21swapper/209:31:092
14125991510,3cyclictest16749-21wc08:20:192
1412599150,14cyclictest5437-21sensors10:25:232
1412399153,11cyclictest1615-21/usr/sbin/munin11:40:211
1411999150,1cyclictest0-21swapper/009:24:390
1412899149,2cyclictest0-21swapper/311:43:223
1412899147,2cyclictest141rcu_preempt09:10:253
1412899143,10cyclictest0-21swapper/312:25:323
1412899142,10cyclictest39-21ksoftirqd/312:31:583
14128991410,1cyclictest39-21ksoftirqd/309:48:323
1412899140,3cyclictest0-21swapper/308:55:283
1412899140,1cyclictest141rcu_preempt10:02:323
1412899140,1cyclictest0-21swapper/310:52:523
1412599142,10cyclictest29814-21/usr/sbin/munin10:20:262
14125991413,1cyclictest0-21swapper/210:03:022
14125991412,1cyclictest0-21swapper/209:17:032
14125991410,3cyclictest13659-21grep10:50:142
14125991410,2cyclictest2075-21sed11:40:252
14125991410,2cyclictest1616-21grep12:00:152
1412599140,14cyclictest0-21swapper/211:52:232
1412399149,1cyclictest0-21swapper/109:19:291
1412399148,5cyclictest1235-21ntpd11:55:201
1412399148,5cyclictest0-21swapper/109:54:401
1412399145,9cyclictest24258-21irqstats11:15:181
1412399142,2cyclictest10397-21apache210:25:111
14123991412,2cyclictest25-21ksoftirqd/109:14:021
14123991411,2cyclictest17627-21uniq07:15:181
1412399140,1cyclictest0-21swapper/110:53:561
1412399140,12cyclictest0-21swapper/111:31:441
1411999142,2cyclictest0-21swapper/010:19:080
14119991413,1cyclictest0-21swapper/012:09:310
14119991410,3cyclictest10858-21cat08:10:000
1411999140,13cyclictest0-21swapper/011:34:450
1412899138,1cyclictest39-21ksoftirqd/309:54:203
1412899135,4cyclictest39-21ksoftirqd/310:35:253
14128991312,1cyclictest0-21swapper/312:00:383
14128991312,1cyclictest0-21swapper/311:51:503
14128991312,1cyclictest0-21swapper/309:23:253
1412899130,1cyclictest141rcu_preempt09:37:423
1412899130,11cyclictest0-21swapper/311:49:283
1412599139,3cyclictest8805-21tr08:05:152
1412599139,3cyclictest8602-21cut11:25:192
1412599139,3cyclictest28898-21cut07:40:142
1412599139,3cyclictest19098-21sendmail-msp07:20:012
1412599139,3cyclictest11048-21grep08:10:142
1412599139,2cyclictest2888-21tail09:45:192
1412599139,2cyclictest17050-21wc11:30:192
14125991310,3cyclictest10397-21apache212:37:552
14125991310,2cyclictest9782-21basename12:05:132
14125991310,2cyclictest30088-21seq12:16:442
14125991310,2cyclictest23703-21wc10:55:272
14125991310,2cyclictest22621-21apt08:35:002
14125991310,2cyclictest18356-21tr12:10:142
14125991310,2cyclictest14905-21grep07:10:162
1412599130,1cyclictest0-21swapper/211:03:222
1412399139,4cyclictest2483-21ssh12:39:161
1412399139,3cyclictest30304-21cat07:40:251
1412399131,6cyclictest27830-21sshd08:45:101
14123991311,1cyclictest7076-21diskmemload09:48:321
1412399130,4cyclictest0-21swapper/110:06:511
1411999139,3cyclictest8809-21cut09:10:180
1411999139,3cyclictest28064-21wc10:20:150
1411999139,2cyclictest765-21munin-plugin-st12:00:000
1411999139,2cyclictest29098-21tr07:40:140
1411999131,7cyclictest20251-21munin-run08:30:000
14119991311,2cyclictest13-21ksoftirqd/011:36:130
14119991310,2cyclictest24365-21grep07:30:150
14119991310,2cyclictest11657-21grep08:10:190
1411999130,13cyclictest0-21swapper/010:02:200
1412899129,2cyclictest9366-21sort11:25:263
1412899129,2cyclictest31199-21ssh10:21:283
1412899129,2cyclictest2896-21tr12:20:133
1412899129,2cyclictest10913-21sed09:30:253
1412899126,6cyclictest8872-21taskset10:27:543
1412899125,3cyclictest225-21systemd-journal08:20:113
1412899124,6cyclictest7076-21diskmemload12:06:573
1412899123,5cyclictest225-21systemd-journal08:53:043
14128991210,1cyclictest23860-21sed08:35:143
1412899120,9cyclictest0-21swapper/312:12:493
1412899120,1cyclictest0-21swapper/312:39:283
1412599129,2cyclictest4033-21wc09:00:282
1412599129,2cyclictest21458-21sort10:35:142
1412599129,2cyclictest1054-21grep11:20:282
1412599128,3cyclictest27301-21sed11:55:262
1412599128,3cyclictest18054-21chrt09:35:172
1412599127,4cyclictest55650irq/127-enp2s0-09:24:182
1412599121,6cyclictest5930-21gltestperf10:45:162
1412599120,3cyclictest0-21swapper/212:21:432
1412399129,3cyclictest7487-21ssh12:03:431
1412399129,2cyclictest9327-21expr08:05:161
1412399127,4cyclictest21206-21taskset07:22:001
1412399123,8cyclictest8657-21ssh10:46:391
1412399122,8cyclictest31573-21/usr/sbin/munin07:45:141
14123991211,1cyclictest0-21swapper/110:14:051
14123991211,1cyclictest0-21swapper/110:14:051
14123991210,1cyclictest21147-21grep08:30:161
1412399120,12cyclictest0-21swapper/109:39:421
1412399120,11cyclictest9048-21kworker/u8:012:21:241
1412399120,11cyclictest19230-21cat08:25:201
1411999129,2cyclictest5473-21head07:55:270
1411999129,2cyclictest21505-21grep10:35:140
1411999129,2cyclictest21106-21grep08:30:160
1411999129,2cyclictest20216-21chrt12:30:150
1411999129,2cyclictest20124-21head07:20:160
1411999129,2cyclictest13855-21tr10:50:140
1411999128,3cyclictest24904-21basename11:55:010
1411999128,3cyclictest22761-21grep10:55:180
1411999128,3cyclictest1637-21tr07:50:140
1411999128,2cyclictest4106-21grep10:25:140
1411999126,2cyclictest225-21systemd-journal07:39:590
1411999122,9cyclictest0-21swapper/009:50:360
1412899119,2cyclictest1682-21grep09:25:213
1412899119,1cyclictest8540-21bc08:05:123
1412899119,1cyclictest43382sleep309:00:353
1412899119,1cyclictest3502-21cron07:55:003
1412899119,1cyclictest31672-21tr07:45:143
1412899119,1cyclictest29499-21grep08:45:263
1412899119,1cyclictest28157-21head07:35:253
1412899119,1cyclictest26166-21cut08:40:173
1412899119,1cyclictest24850-21cut07:30:163
1412899119,1cyclictest13953-21cut08:15:163
1412899119,1cyclictest11189-21cat08:10:143
1412899118,2cyclictest25987-21tr09:40:143
1412899110,10cyclictest26312-21sh11:35:293
1412599118,2cyclictest6539-21ssh11:05:142
1412599118,2cyclictest5716-21grep09:05:172
1412599118,2cyclictest4730-21ssh09:27:192
1412599118,2cyclictest3379-21fschecks_time10:05:162
1412599118,2cyclictest31683-21tr07:45:142
1412599118,2cyclictest31212-21cat08:50:192
1412599118,2cyclictest25559-21ntpq11:35:222
1412599118,2cyclictest24527-21expr07:30:142
1412599118,2cyclictest23359-21expr08:35:112
1412599118,2cyclictest22952-21grep07:25:222
1412599118,2cyclictest21497-21grep12:30:242
1412599118,2cyclictest14030-21sort08:15:162
1412599118,2cyclictest13018-21grep10:10:252
1412599118,2cyclictest13018-21grep10:10:252
1412599118,2cyclictest11496-21expr12:25:152
1412399119,2cyclictest3344-21taskset07:53:011
1412399119,2cyclictest30485-21taskset10:20:551
1412399119,2cyclictest14823-21cat08:15:251
1412399119,1cyclictest27360-21taskset08:42:051
1412399119,1cyclictest24395-21cat07:30:151
1412399119,1cyclictest16580-21rm10:32:201
1412399119,1cyclictest16026-21grep07:10:271
1412399119,1cyclictest1226-21cat08:55:191
1412399118,2cyclictest7158-21wc08:00:191
1412399118,2cyclictest18001-21sshd12:10:101
1412399118,2cyclictest17577-21taskset08:21:251
1412399118,2cyclictest12631-21grep08:10:291
14123991110,1cyclictest0-21swapper/112:16:091
1412399110,10cyclictest23089-21chrt08:35:101
1411999118,2cyclictest8311-21ssh11:25:170
1411999118,2cyclictest6771-21chrt09:06:170
1411999118,2cyclictest6423-21ssh11:05:110
1411999118,2cyclictest6105-21cat10:45:190
1411999118,2cyclictest3600-21ssh09:26:260
1411999118,2cyclictest3495-21cat12:20:160
1411999118,2cyclictest32503-21smartctl11:20:180
1411999118,2cyclictest30886-21grep08:50:170
1411999118,2cyclictest29662-21taskset08:46:100
1411999118,2cyclictest29119-21ssh10:40:100
1411999118,2cyclictest27315-21ssh09:40:230
1411999118,2cyclictest26578-21grep12:15:140
1411999118,2cyclictest2625-21ssh11:02:340
1411999118,2cyclictest23767-21cut08:35:140
1411999118,2cyclictest2013-21grep11:40:250
1411999118,2cyclictest17710-21ssh11:11:050
1411999118,2cyclictest1686-21ssh12:38:390
1411999118,2cyclictest16821-21grep09:15:160
1411999118,2cyclictest1635-21cat08:55:250
1411999118,2cyclictest1474-21ssh09:45:050
1411999118,2cyclictest12033-21ssh10:10:170
1411999118,2cyclictest12033-21ssh10:10:170
1411999114,3cyclictest225-21systemd-journal12:00:220
1411999113,4cyclictest225-21systemd-journal08:25:010
1411999110,10cyclictest2825-21cut09:00:170
1411999110,10cyclictest12678-21sensors12:25:250
1412899109,1cyclictest8021-21taskset08:01:513
1412899109,1cyclictest22275-21seq08:30:483
1412899108,1cyclictest5240-21tr09:05:153
1412899108,1cyclictest3829-21cut07:55:143
1412899108,1cyclictest27084-21wc11:55:253
1412899108,1cyclictest18846-21grep08:25:173
1412899108,1cyclictest17210-21tr07:15:153
1412899107,2cyclictest15089-21grep07:10:193
1412899100,10cyclictest21325-21taskset07:23:313
1412599108,1cyclictest210902sleep207:20:292
1412599108,1cyclictest199712sleep208:26:412
1412599107,2cyclictest4710-21uname07:55:202
1412599107,2cyclictest32620-21chrt08:55:132
1412599107,2cyclictest3041-21sed07:50:262
1412599107,2cyclictest27854-21cat08:45:122
1412599107,2cyclictest27156-21cut07:35:152
1412599102,4cyclictest225-21systemd-journal08:40:002
1412399108,2cyclictest5313-21expr09:05:151
1412399108,2cyclictest26766-21seq07:35:131
1412399108,1cyclictest4541-21chrt09:03:111
1412399108,1cyclictest3741-21sshd07:55:101
1412399107,2cyclictest30445-21cat08:50:141
1411999108,1cyclictest16946-21apache_accesses07:15:130
1411999108,1cyclictest15869-21ls07:10:270
1411999107,2cyclictest32221-21head07:45:170
141289998,1cyclictest30419-21cat07:40:263
141259997,1cyclictest80172sleep208:01:482
141199997,1cyclictest8034-21taskset08:02:000
141199985,2cyclictest26210-21grep08:40:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional