You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 07:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 02, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302728976,9sleep20-21swapper/219:05:592
319528774,9sleep00-21swapper/019:08:080
314628159,6sleep10-21swapper/119:07:301
332827757,15sleep30-21swapper/319:09:533
343399326,22cyclictest32315-21taskset21:30:273
343399272,2cyclictest26337-21sh22:25:183
342399270,7cyclictest0-21swapper/023:03:190
343399252,2cyclictest0-21swapper/323:10:163
343399240,6cyclictest0-21swapper/323:16:123
343399240,4cyclictest28214-21diskmemload23:30:183
342399242,2cyclictest0-21swapper/023:10:160
343399224,1cyclictest0-21swapper/321:48:073
342899220,21cyclictest19835-21kworker/1:022:25:241
343399211,1cyclictest30652-21ssh23:06:273
343399210,15cyclictest0-21swapper/321:51:013
3428992117,3cyclictest25-21ksoftirqd/119:25:101
342899210,2cyclictest4910-21ps19:10:221
342899210,21cyclictest0-21swapper/123:58:391
343399202,1cyclictest3213-21runrttasks22:11:093
343399200,2cyclictest30153-21kworker/u8:222:31:273
343399200,1cyclictest0-21swapper/323:41:053
343399200,1cyclictest0-21swapper/321:27:143
343399194,3cyclictest14108-21/usr/sbin/munin20:40:193
343399194,2cyclictest28214-21diskmemload22:55:213
343399194,2cyclictest28214-21diskmemload22:55:213
343399193,4cyclictest24176-21ssh23:22:083
343399193,2cyclictest0-21swapper/322:41:453
343399190,3cyclictest0-21swapper/323:26:103
343399190,1cyclictest0-21swapper/321:12:533
3428991918,1cyclictest0-21swapper/123:29:211
3428991917,1cyclictest0-21swapper/123:21:481
342899190,19cyclictest0-21swapper/121:15:261
342899190,18cyclictest0-21swapper/121:46:591
342899190,0cyclictest0-21swapper/123:01:331
342399194,3cyclictest25611-21memory22:05:220
58292180,2sleep10-21swapper/122:51:381
343399185,2cyclictest6976-21tr00:10:133
343399182,2cyclictest28214-21diskmemload21:20:193
343399180,1cyclictest0-21swapper/323:35:383
343099180,18cyclictest0-21swapper/223:03:382
342899181,1cyclictest14617-21needreboot21:20:201
342899180,3cyclictest17956-21grep22:00:261
342899180,1cyclictest0-21swapper/121:58:151
342899180,18cyclictest0-21swapper/122:55:141
342899180,18cyclictest0-21swapper/122:55:131
342899180,18cyclictest0-21swapper/100:06:011
3423991810,6cyclictest30176-21grep23:25:250
343399174,1cyclictest28214-21diskmemload22:24:293
343399172,6cyclictest9593-21/usr/sbin/munin22:35:153
343399172,1cyclictest6391-21apt-get21:35:133
343399171,1cyclictest0-21swapper/323:45:043
3430991715,1cyclictest0-21swapper/222:40:262
343099170,3cyclictest0-21swapper/221:23:432
343099170,17cyclictest0-21swapper/223:50:522
343099170,17cyclictest0-21swapper/221:47:362
343099170,17cyclictest0-21swapper/221:41:312
343099170,16cyclictest0-21swapper/223:14:012
3428991715,2cyclictest0-21swapper/123:11:121
342899170,3cyclictest0-21swapper/121:31:181
342899170,16cyclictest0-21swapper/122:40:131
342899170,15cyclictest0-21swapper/122:30:451
342399179,2cyclictest623-21grep21:50:210
342399179,2cyclictest18283-21grep19:40:130
342399176,2cyclictest452-21systemd-logind00:08:570
342399173,8cyclictest12253-21diskstats23:15:170
342399172,7cyclictest3944-21unixbench_singl20:15:260
93522160,3sleep2311rcuc/219:20:172
343399168,7cyclictest6447-21tail23:50:203
343399166,3cyclictest39-21ksoftirqd/300:09:583
343399164,2cyclictest12906-1kworker/u9:023:58:523
3433991610,5cyclictest0-21swapper/300:00:163
343399160,4cyclictest0-21swapper/300:27:043
343399160,3cyclictest0-21swapper/322:18:463
343399160,2cyclictest0-21swapper/300:33:023
343099163,6cyclictest23633-21apt-get00:20:122
343099161,15cyclictest0-21swapper/219:45:142
343099160,16cyclictest0-21swapper/223:18:212
343099160,16cyclictest0-21swapper/222:01:192
342899165,6cyclictest22761-21apt-get19:50:141
342899162,1cyclictest141rcu_preempt22:35:531
3428991615,1cyclictest0-21swapper/120:17:091
342899160,1cyclictest0-21swapper/100:31:431
342399169,6cyclictest25801-21grep19:55:170
342399169,6cyclictest15622-21tr19:35:110
342399169,2cyclictest9768-21ssh00:11:100
342399169,2cyclictest17579-21cat22:40:000
342399168,7cyclictest6856-21sed21:15:270
342399168,2cyclictest3135-21grep22:50:170
342399168,2cyclictest29831-21ssh23:05:490
342399162,5cyclictest7957-21aten2_r7power_e00:30:150
3423991615,1cyclictest13-21ksoftirqd/022:30:170
3423991610,5cyclictest7838-21wc21:35:210
342399160,10cyclictest0-21swapper/023:31:160
343399158,6cyclictest8054-21wc20:25:193
343399158,6cyclictest19088-21cat19:40:183
343399158,6cyclictest17820-21ssh22:00:253
343399158,6cyclictest17655-21ssh21:41:053
343399153,1cyclictest28214-21diskmemload23:04:173
343399152,1cyclictest28214-21diskmemload22:48:153
3433991510,4cyclictest13231-21wc20:35:263
343399150,3cyclictest0-21swapper/300:38:443
343099155,9cyclictest0-21swapper/200:31:542
343099150,15cyclictest0-21swapper/222:30:552
343099150,15cyclictest0-21swapper/200:11:032
342899152,0cyclictest141rcu_preempt22:45:241
342899150,0cyclictest0-21swapper/100:23:321
342399159,3cyclictest7446-21ssh00:30:000
342399159,2cyclictest24838-21tr21:00:200
342399158,2cyclictest20595-21ssh23:59:130
342399158,2cyclictest16242-21cron20:45:000
342399158,2cyclictest14348-21rm21:20:180
342399156,2cyclictest6660-21wc23:50:220
342399150,6cyclictest0-21swapper/021:33:360
342399150,2cyclictest0-21swapper/000:23:000
343399148,5cyclictest4576-21ssh21:15:023
343399148,5cyclictest4149-21grep19:10:163
343399147,6cyclictest8345-21cut21:55:163
343399147,6cyclictest6222-21ssh22:51:553
343399147,6cyclictest4307-21chrt20:19:443
343399147,6cyclictest32639-21expr20:10:133
343399147,6cyclictest10538-21sh19:25:003
343399147,2cyclictest19042-21cut20:50:133
343399145,2cyclictest26876-21cut21:05:163
343399145,2cyclictest22554-21tr20:55:223
343399142,3cyclictest28214-21diskmemload00:15:013
343399142,3cyclictest225-21systemd-journal20:25:003
343399142,2cyclictest225-21systemd-journal19:40:003
343399141,3cyclictest225-21systemd-journal19:30:243
3433991413,1cyclictest0-21swapper/321:00:123
343099146,4cyclictest225-21systemd-journal00:10:002
3430991410,3cyclictest6801-21cat20:25:002
343099140,14cyclictest0-21swapper/221:12:562
343099140,13cyclictest0-21swapper/200:38:092
3428991411,2cyclictest2764-21awk23:10:001
3428991410,3cyclictest1468-21grep20:10:241
342399148,1cyclictest6131-21cut19:15:130
342399148,1cyclictest236902sleep000:39:430
342399148,1cyclictest16662-21ntpq00:15:210
342399147,2cyclictest31675-21chrt20:05:550
342399147,2cyclictest29108-21grep21:10:150
342399147,2cyclictest28083-21ssh21:47:440
342399147,2cyclictest18500-21taskset20:47:470
342399147,2cyclictest13261-21grep23:35:170
342399147,1cyclictest667-21cat20:10:160
342399147,1cyclictest16263-21gzip21:40:220
342399146,3cyclictest11579-21grep19:25:160
342399143,5cyclictest17014-21apache222:21:400
3423991410,3cyclictest26376-21tr21:05:140
342399140,9cyclictest23003-21ssh21:25:180
343399137,6cyclictest0-21swapper/322:05:313
343399136,2cyclictest23789-21sort19:50:213
343399130,4cyclictest225-21systemd-journal20:35:003
343399130,2cyclictest0-21swapper/319:55:143
343099139,3cyclictest7295-21wc19:15:212
343099139,3cyclictest28439-21cut23:25:132
343099134,7cyclictest7634-21needreboot21:35:192
343099133,2cyclictest225-21systemd-journal21:00:002
3430991311,2cyclictest32-21ksoftirqd/223:39:522
3430991310,2cyclictest9202-21df22:15:162
3430991310,2cyclictest5959-21wc19:15:002
3430991310,2cyclictest29581-21cron21:30:002
3430991310,2cyclictest27142-21cat22:25:252
3430991310,2cyclictest24012-21wc19:50:232
3430991310,2cyclictest12490-21grep19:25:252
343099130,13cyclictest0-21swapper/200:28:122
3428991310,2cyclictest7096-21cron21:55:011
3428991310,2cyclictest26347-21grep19:55:221
3428991310,2cyclictest17318-21grep19:35:231
342399138,1cyclictest29626-21ssh22:27:100
342399138,1cyclictest232522sleep020:57:520
342399137,2cyclictest20007-21ssh23:20:120
342399137,1cyclictest7032-21cut20:25:130
342399135,7cyclictest15296-21munin-run19:35:000
342399132,8cyclictest5794-21sed19:15:000
3423991310,2cyclictest8759-21cut22:15:150
3423991310,2cyclictest8128-21cron22:15:000
342399130,9cyclictest22767-21ssh22:42:330
343399122,3cyclictest28774-21wc20:00:223
343399121,5cyclictest12596-21taskset19:26:033
343399121,3cyclictest225-21systemd-journal00:20:003
343399121,2cyclictest225-21systemd-journal19:15:243
343399120,3cyclictest225-21systemd-journal20:05:063
343099129,2cyclictest30157-21head23:45:162
343099129,2cyclictest29147-21grep20:00:262
343099129,2cyclictest28190-21cron21:10:002
343099129,2cyclictest26359-21tr19:55:222
343099129,2cyclictest2506-21tr20:15:142
343099129,2cyclictest24793-21cut22:05:182
343099129,2cyclictest24206-21grep21:00:152
343099129,2cyclictest18746-21head19:40:162
343099129,2cyclictest1835-21cron22:50:002
343099129,2cyclictest17809-21head20:45:202
343099129,2cyclictest16146-21grep00:15:172
343099129,2cyclictest12678-21cut20:35:192
343099129,2cyclictest11473-21cron20:35:002
3430991211,1cyclictest55650irq/127-enp2s0-22:10:152
3430991211,1cyclictest3418-21cyclictest21:57:372
3430991210,1cyclictest7306-21cat21:55:012
343099120,11cyclictest2773-21awk23:10:002
342899129,2cyclictest9191-21head19:20:151
342899129,2cyclictest7525-21grep00:10:161
342899129,2cyclictest28807-21sh23:44:591
342899129,2cyclictest26835-21grep21:05:161
342899129,2cyclictest25380-21grep21:00:261
342899129,2cyclictest24735-21grep22:05:181
342899129,2cyclictest11853-21tr20:35:141
342899122,9cyclictest17803-21runrttasks23:18:411
342899122,9cyclictest0-21swapper/100:35:451
3428991210,1cyclictest25-21ksoftirqd/122:19:381
342899120,1cyclictest0-21swapper/123:51:391
342899120,11cyclictest0-21swapper/100:28:411
342399129,2cyclictest16772-21head22:00:160
342399129,2cyclictest10918-21sort20:30:250
342399128,2cyclictest13459-21grep20:35:270
343399118,2cyclictest17678-21grep20:45:183
343399110,4cyclictest20997-21tune2fs19:45:173
343099118,2cyclictest21387-21grep23:20:212
343099118,2cyclictest19370-21ls20:50:122
343099118,2cyclictest17921-21grep22:20:172
343099118,2cyclictest15547-21apache_accesses19:35:092
343099118,2cyclictest14607-21grep23:55:172
343099118,2cyclictest12433-21ssh22:36:042
343099118,2cyclictest12271-21ssh22:55:252
343099118,2cyclictest12271-21ssh22:55:252
343099118,2cyclictest11131-21sh23:34:172
343099117,3cyclictest22149-21ssh00:00:132
343099117,3cyclictest15835-21grep20:40:262
343099110,1cyclictest0-21swapper/222:53:172
343099110,1cyclictest0-21swapper/221:15:142
343099110,11cyclictest0-21swapper/221:30:152
343099110,10cyclictest24309-21ssh23:41:292
342899119,1cyclictest29789-21grep21:10:171
342899118,2cyclictest981-21grep22:10:161
342899118,2cyclictest8750-21sh21:35:411
342899118,2cyclictest5371-21tr20:20:171
342899118,2cyclictest29623-21sh20:05:001
342899118,2cyclictest27990-21ssh00:03:281
342899118,2cyclictest20268-21find20:50:211
342899118,2cyclictest17020-21ssh22:20:121
342899118,2cyclictest14996-21taskset19:31:231
342899117,3cyclictest21086-21grep19:45:171
342899117,3cyclictest19968-21rm00:17:301
342899117,3cyclictest19433-21ssh21:42:291
342899117,2cyclictest7144-21ssh23:31:091
342899115,4cyclictest23449-21sleep21:00:001
342899113,3cyclictest13922-21pmu-power23:35:221
342899110,1cyclictest0-21swapper/121:25:081
342899110,11cyclictest0-21swapper/123:46:441
342399118,2cyclictest29034-21ssh23:45:010
342399118,2cyclictest28479-21ssh22:45:300
342399118,2cyclictest23258-21ssh23:40:390
342399118,2cyclictest21825-21cat19:45:250
342399118,2cyclictest20821-21taskset20:52:080
342399118,2cyclictest11438-21grep22:55:170
342399118,2cyclictest11438-21grep22:55:170
342399117,2cyclictest27756-21tr20:00:150
342399112,7cyclictest10549-21ls19:25:000
342399110,2cyclictest0-21swapper/021:58:320
342399110,10cyclictest5271-21cut20:20:170
343099108,1cyclictest30041-21egrep20:05:152
343099108,1cyclictest1024-21open_inodes20:10:202
343099107,2cyclictest8833-21taskset20:26:362
343099107,2cyclictest14111-21kernelversion19:30:182
342899108,1cyclictest6154-21hostname19:15:131
342899108,1cyclictest19878-21seq19:42:071
342899108,1cyclictest10919-21cat20:30:251
342899107,2cyclictest7715-21head20:25:161
342899107,2cyclictest17290-21head20:45:161
342899106,3cyclictest30107-21cat20:05:151
342899105,4cyclictest15102-21expr20:40:181
342399108,1cyclictest22961-21grep19:50:160
342399107,2cyclictest21887-21ssh00:00:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional