You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 22:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 01, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3211529380,9sleep20-21swapper/207:07:042
3232629279,9sleep00-21swapper/007:09:460
3221328564,6sleep30-21swapper/307:08:213
3210928366,13sleep10-21swapper/107:06:581
215302530,0sleep20-21swapper/212:00:472
3243099260,24cyclictest0-21swapper/209:28:192
3243099240,24cyclictest0-21swapper/211:19:272
3243099240,23cyclictest0-21swapper/209:47:532
70182230,1sleep30-21swapper/311:33:233
3243099230,2cyclictest0-21swapper/211:44:172
32430992219,2cyclictest0-21swapper/212:05:342
3243099220,21cyclictest0-21swapper/211:25:132
3242099220,20cyclictest0-21swapper/012:34:500
3243099210,21cyclictest0-21swapper/211:56:052
3243099210,21cyclictest0-21swapper/211:51:122
3242599210,2cyclictest0-21swapper/109:40:151
3242099211,8cyclictest0-21swapper/011:55:130
3242099210,4cyclictest0-21swapper/009:25:550
271532210,0sleep30-21swapper/309:10:213
3243099200,20cyclictest0-21swapper/211:21:212
3242599201,13cyclictest1957-21munin-run11:50:001
3242599200,2cyclictest0-21swapper/109:58:001
3242099196,6cyclictest25213-21diskmemload10:50:000
32420991919,0cyclictest0-21swapper/009:37:520
32434991817,1cyclictest6083-21kworker/3:110:28:133
3243099180,18cyclictest0-21swapper/211:31:382
3243099180,18cyclictest0-21swapper/210:39:442
3242599180,10cyclictest13524-21aten2_r7power_c10:20:121
3242099181,17cyclictest0-21swapper/011:11:450
3242099180,18cyclictest0-21swapper/012:19:590
32434991716,1cyclictest0-21swapper/311:45:463
3243499171,15cyclictest0-21swapper/307:10:123
3243499170,17cyclictest0-21swapper/309:25:203
3243499170,16cyclictest0-21swapper/312:27:333
3243499170,15cyclictest0-21swapper/311:15:373
32430991714,2cyclictest31520-21apache_volume10:50:122
3243099170,17cyclictest0-21swapper/210:31:572
3243099170,15cyclictest0-21swapper/210:00:222
3242599173,3cyclictest141rcu_preempt10:26:371
32425991716,1cyclictest25432-21kworker/1:212:30:331
32425991716,1cyclictest0-21swapper/112:05:171
32425991716,1cyclictest0-21swapper/111:27:391
32425991715,2cyclictest4306-21kworker/1:010:44:331
3242599170,17cyclictest0-21swapper/112:23:201
32420991717,0cyclictest0-21swapper/010:44:270
32420991716,1cyclictest13-21ksoftirqd/010:51:550
3242099170,2cyclictest0-21swapper/010:35:430
86212160,0chrt0-21swapper/309:57:263
32434991615,1cyclictest0-21swapper/308:31:233
3243099166,9cyclictest0-21swapper/208:35:282
3242599165,4cyclictest225-21systemd-journal11:59:591
3242599161,4cyclictest25213-21diskmemload11:15:221
32420991613,2cyclictest6067-21users09:55:280
32420991613,2cyclictest29115-21apache_processe10:10:120
3242099160,1cyclictest0-21swapper/011:49:060
3243499152,3cyclictest22669-21unixbench-2d10:05:263
32434991510,4cyclictest31022-21perf10:50:003
3243499150,13cyclictest0-21swapper/312:06:083
3243099150,2cyclictest0-21swapper/210:26:122
3243099150,0cyclictest0-21swapper/211:48:562
3242599151,1cyclictest25213-21diskmemload10:53:451
3242599150,10cyclictest49-21kcompactd007:14:301
3242099150,1cyclictest0-21swapper/011:32:070
3243499148,1cyclictest0-21swapper/309:43:053
32434991411,1cyclictest25213-21diskmemload12:15:173
3243499140,8cyclictest0-21swapper/307:50:143
3243099149,3cyclictest21654-21cut10:05:172
3243099145,6cyclictest29673-21apache209:54:012
3243099145,6cyclictest29673-21apache209:54:012
32430991412,1cyclictest0-21swapper/209:55:142
32430991411,2cyclictest25225-21perf09:10:002
3242599149,4cyclictest25738-21sshd09:29:341
3242599148,5cyclictest9841-21taskset07:30:001
3242599143,3cyclictest19106-21apt-get12:00:131
32425991411,2cyclictest23184-21apache_volume10:45:111
32425991411,2cyclictest15840-21apache_volume11:00:131
3242099142,7cyclictest0-21swapper/011:01:200
32420991410,3cyclictest20044-21sh11:40:270
32420991410,3cyclictest1218-21ssh12:08:400
32420991410,2cyclictest3747-21cat09:55:000
32420991410,2cyclictest3747-21cat09:55:000
3242099140,14cyclictest0-21swapper/009:16:120
3243499136,3cyclictest25662-21runrttasks08:00:253
3243499132,3cyclictest29069-21/usr/sbin/munin10:10:163
3243499132,11cyclictest49-21kcompactd007:20:573
3243499132,10cyclictest25849-21sensors11:05:243
3243499131,10cyclictest0-21swapper/308:15:153
32434991310,2cyclictest5440-21grep10:15:153
32434991310,2cyclictest21466-21head09:00:153
32434991310,2cyclictest21363-21cron10:25:003
32434991310,2cyclictest14614-21sh10:40:113
32434991310,2cyclictest13100-21grep07:35:183
3243499130,13cyclictest0-21swapper/311:29:223
3243099139,3cyclictest3640-21grep09:15:262
3243099139,3cyclictest26385-21turbostat.cron08:05:002
3243099139,3cyclictest19221-21cron07:50:002
32430991312,1cyclictest0-21swapper/209:41:132
32430991310,2cyclictest9924-21tr11:35:152
32430991310,2cyclictest4906-21wc07:20:002
32430991310,2cyclictest4430-21apache_volume12:30:132
32430991310,2cyclictest16529-21grep10:40:272
3243099130,12cyclictest25902-21ssh11:05:272
3242599135,8cyclictest38150irq/131-snd_hda11:41:201
3242599135,6cyclictest30689-21/usr/sbin/munin10:10:221
3242599131,9cyclictest516-21gdbus09:05:241
32425991312,1cyclictest0-21swapper/112:15:471
32425991311,1cyclictest7489-21apache_volume10:55:131
32425991310,2cyclictest10690-21grep07:30:171
32425991310,2cyclictest10255-21cat08:35:271
3242599130,9cyclictest0-21swapper/111:52:471
3242599130,13cyclictest0-21swapper/111:36:411
3242099139,3cyclictest16452-21sed07:40:260
3242099132,9cyclictest0-21swapper/011:17:190
32420991310,2cyclictest8041-21grep08:30:270
32420991310,2cyclictest20937-21grep12:20:150
32420991310,2cyclictest20257-21grep07:50:170
32420991310,2cyclictest16881-21apache_processe11:20:120
32420991310,2cyclictest1157-21perf08:20:000
32420991310,2cyclictest0-21swapper/009:47:140
3242099130,1cyclictest0-21swapper/009:22:400
3242099130,12cyclictest9315-21wc10:55:270
3243499129,3cyclictest6083-21kworker/3:109:20:043
3243499129,2cyclictest8286-21uniq07:25:173
3243499129,2cyclictest7048-21cut10:35:193
3243499129,2cyclictest5674-21grep08:25:263
3243499129,2cyclictest24185-21grep09:05:213
3243499129,2cyclictest2366-21head08:20:163
3243499129,2cyclictest16888-21cat07:45:003
3243499129,2cyclictest1477-21ssh09:15:013
3243499128,2cyclictest11877-21ssh10:57:193
32434991210,1cyclictest17818-21tune2fs07:45:183
3243499120,7cyclictest0-21swapper/312:35:133
3243499120,5cyclictest0-21swapper/312:00:133
3243099129,2cyclictest9361-21grep10:55:262
3243099129,2cyclictest5189-21grep12:10:262
3243099129,2cyclictest4566-21cat09:35:262
3243099129,2cyclictest4057-21cut08:25:142
3243099129,2cyclictest404-21grep11:10:152
3243099129,2cyclictest31095-21tail10:10:262
3243099129,2cyclictest30163-21cut08:10:192
3243099129,2cyclictest28471-21wc09:30:262
3243099129,2cyclictest26678-21cut08:05:142
3243099129,2cyclictest26678-21cut08:05:132
3243099129,2cyclictest19719-21tr07:50:152
3243099129,2cyclictest19667-21chrt11:01:522
3243099129,2cyclictest13202-21chrt10:20:022
3243099129,2cyclictest12392-21grep07:35:152
3243099129,2cyclictest12104-21apt07:35:002
3243099128,3cyclictest19094-21sed08:55:172
3243099128,3cyclictest13806-21tr08:45:132
32430991212,0cyclictest0-21swapper/208:40:132
3243099120,12cyclictest0-21swapper/209:22:532
3242599129,2cyclictest5800-21sort10:15:161
3242599129,2cyclictest4728-21cron07:20:001
3242599129,2cyclictest31891-21ssh10:30:231
3242599129,2cyclictest26966-21grep09:10:201
3242599129,2cyclictest21186-21grep07:50:271
3242599129,2cyclictest18382-21wc07:45:231
3242599129,2cyclictest16660-21tail08:50:171
3242599129,2cyclictest15974-21head07:40:221
3242599129,2cyclictest13415-21wc07:35:201
3242599129,2cyclictest11445-21tr08:40:141
3242599128,3cyclictest2751-21grep08:20:191
3242599127,4cyclictest13248-21ssh12:35:151
3242599126,2cyclictest21548-21ssh09:45:281
32425991210,1cyclictest7120-21head10:35:191
32425991210,1cyclictest0-21swapper/110:02:471
3242599120,11cyclictest29179-21grep12:25:161
3242599120,11cyclictest28367-21tr09:50:151
3242599120,11cyclictest28367-21tr09:50:151
3242599120,11cyclictest20639-21wc09:00:001
3242099129,2cyclictest4736-21head08:25:180
3242099129,2cyclictest27222-21wc09:10:210
3242099129,2cyclictest24864-21grep08:00:170
3242099129,2cyclictest23560-21grep07:55:240
3242099129,2cyclictest19079-21grep08:55:170
3242099129,2cyclictest16222-21grep08:50:160
3242099129,2cyclictest11950-21sed08:40:170
3242099129,1cyclictest0-21swapper/009:33:350
3242099128,3cyclictest29673-21apache211:39:540
3242099128,3cyclictest13072-21sed09:40:240
3242099125,4cyclictest0-21swapper/012:38:540
32420991211,1cyclictest0-21swapper/010:15:390
32420991210,1cyclictest13624-21memory10:00:200
3242099120,11cyclictest13728-21tr10:20:120
3243499119,1cyclictest28412sleep309:35:143
3243499119,1cyclictest10716-21head07:30:173
3243499118,2cyclictest9410-21ssh11:35:053
3243499118,2cyclictest5709-21ntpq12:30:213
3243499118,2cyclictest5195-21cat12:10:263
3243499118,2cyclictest32581-21chrt10:50:193
3243499118,2cyclictest23818-21ssh12:21:463
3243499118,2cyclictest2201-21sh11:50:023
3243499118,2cyclictest19768-21cat08:55:263
3243499118,2cyclictest19559-21ssh09:45:143
3243499118,2cyclictest17342-21wc11:00:233
3243499118,2cyclictest13330-21basename08:44:593
3243499118,2cyclictest11874-21cat11:55:193
3243499117,3cyclictest19118-21ssh10:04:063
3243099118,2cyclictest8232-21sort07:25:162
3243099118,2cyclictest7133-21chrt08:30:172
3243099118,2cyclictest29294-21grep12:25:182
3243099118,2cyclictest27431-21ssh10:47:112
3243099118,2cyclictest26967-21head09:10:202
3243099118,2cyclictest2497-21date07:15:002
3243099118,2cyclictest23184-21wc07:55:202
3243099118,2cyclictest23144-21ssh12:21:152
3243099118,2cyclictest17712-21sh12:37:462
3243099118,2cyclictest17550-21ls08:50:272
3243099118,2cyclictest1749-21grep08:20:142
3243099118,2cyclictest15979-21ssh10:20:482
3243099118,2cyclictest15784-21ssh12:17:012
3243099118,2cyclictest1133-21sh08:20:002
3243099117,3cyclictest16395-21tail07:40:262
3243099113,5cyclictest0-21swapper/209:00:112
3242599119,1cyclictest7539-21grep08:30:211
3242599118,2cyclictest5670-21tail08:25:271
3242599118,2cyclictest4439-21ssh12:10:191
3242599118,2cyclictest29249-21tr08:10:141
3242599118,2cyclictest29222-21ssh09:30:461
3242599118,2cyclictest26937-21ssh11:05:591
3242599118,2cyclictest2587-21ssh09:35:041
3242599118,2cyclictest25013-21sed08:00:171
3242599118,2cyclictest22482-21cat10:05:261
3242599118,2cyclictest2207-21grep11:10:271
3242599118,2cyclictest21856-21seq07:55:091
3242599118,2cyclictest21071-21expr09:00:141
3242599118,2cyclictest1550-21tr11:30:161
3242599118,2cyclictest10053-21ssh09:20:101
3242599117,3cyclictest2845-21latency09:15:191
3242599117,3cyclictest18795-21tail11:20:261
3242099118,2cyclictest9046-21grep08:35:160
3242099118,2cyclictest8637-21taskset07:25:190
3242099118,2cyclictest8224-21sh12:12:380
3242099118,2cyclictest6232-21ssh11:51:510
3242099118,2cyclictest6003-21grep07:20:180
3242099118,2cyclictest4726-21cron07:20:000
3242099118,2cyclictest30718-21ssh11:08:530
3242099118,2cyclictest30131-21wc08:10:190
3242099118,2cyclictest29785-21nfs4_client12:25:200
3242099118,2cyclictest27479-21ssh11:25:360
3242099118,2cyclictest2297-21ssh10:32:190
3242099118,2cyclictest21469-21sort10:05:160
3242099118,2cyclictest17401-21tr07:45:150
3242099118,2cyclictest10766-21sed07:30:180
3242099115,2cyclictest225-21systemd-journal09:10:000
3242099112,8cyclictest28774-21perf08:10:000
3242099112,8cyclictest28774-21perf08:10:000
3242099110,10cyclictest255772sleep012:03:570
3243499108,2cyclictest16852-21taskset08:50:193
3243499108,1cyclictest9080-21expr08:35:173
3243499108,1cyclictest4519-21taskset07:17:253
3243499108,1cyclictest347-21aten2_r7power_p11:10:153
3243499108,1cyclictest32417-21ssh10:30:303
3243499108,1cyclictest29319-21cat08:10:153
3243499108,1cyclictest28335-21grep08:05:263
3243499108,1cyclictest28335-21grep08:05:263
3243499108,1cyclictest27876-21ssh09:50:063
3243499108,1cyclictest27876-21ssh09:50:053
3243499108,1cyclictest17884-21taskset11:40:013
3243499108,1cyclictest15855-21cut08:50:003
3243499107,2cyclictest27651-21sed09:30:173
3243499100,9cyclictest227832sleep307:55:163
3243499100,9cyclictest166182sleep311:20:003
3243099108,1cyclictest7271-21taskset07:24:152
3242599107,2cyclictest5380-21egrep07:20:151
3242599107,2cyclictest32400-21cut08:15:211
3242599107,2cyclictest26674-21cut08:05:141
3242599107,2cyclictest26674-21cut08:05:131
3242599104,2cyclictest225-21systemd-journal08:49:591
3242099108,1cyclictest34102sleep008:20:420
3242099108,1cyclictest155722sleep008:48:420
3242099108,1cyclictest13349-21cat07:35:200
3242099107,2cyclictest22194-21grep10:25:150
3242099107,2cyclictest1001-21head07:10:160
3242099102,6cyclictest21473-21gltestperf09:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional