You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 19:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Jun 29, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1036029578,13sleep20-21swapper/207:06:292
1030929464,9sleep30-21swapper/307:05:503
1039328774,9sleep00-21swapper/007:06:550
1038428164,13sleep10-21swapper/107:06:481
1071399270,24cyclictest0-21swapper/109:45:091
1071399261,24cyclictest0-21swapper/112:30:101
1071399231,22cyclictest0-21swapper/112:00:561
1071399230,1cyclictest0-21swapper/111:31:201
1071999220,18cyclictest0-21swapper/310:25:153
10717992221,1cyclictest0-21swapper/210:09:222
1071399210,16cyclictest0-21swapper/109:29:061
1070899210,21cyclictest0-21swapper/009:54:490
1071799190,2cyclictest20965-21diskmemload09:39:062
1070899196,7cyclictest22936-21munin-run11:55:000
1070899190,17cyclictest0-21swapper/011:14:360
10713991813,4cyclictest29349-21idleruntime-cro09:25:001
1071399180,1cyclictest0-21swapper/111:03:281
1071399180,16cyclictest0-21swapper/112:23:231
10719991710,5cyclictest1552-21grep12:20:173
1071799170,17cyclictest0-21swapper/209:56:032
10713991712,4cyclictest38150irq/131-snd_hda10:24:441
1070899177,3cyclictest0-21swapper/010:00:030
1071999160,16cyclictest0-21swapper/312:11:163
1071999160,15cyclictest0-21swapper/309:28:563
1071799161,6cyclictest3947-21diskmemload10:58:442
10717991616,0cyclictest0-21swapper/212:23:212
1071799160,14cyclictest9872-21sh09:51:022
1071799160,13cyclictest0-21swapper/212:15:142
1071399162,13cyclictest8176-21cron10:10:001
1071399160,1cyclictest0-21swapper/111:09:581
1071399160,15cyclictest0-21swapper/111:28:211
10708991615,1cyclictest0-21swapper/010:15:120
1071999150,9cyclictest0-21swapper/311:50:163
10717991512,2cyclictest1423-21sh12:40:002
1071799151,13cyclictest0-21swapper/212:07:062
1071799150,14cyclictest0-21swapper/210:50:022
1071799150,13cyclictest0-21swapper/209:22:572
1071399159,5cyclictest21013-21grep07:30:161
1071399155,9cyclictest27918-21munin-node08:14:391
10713991514,1cyclictest0-21swapper/109:56:491
1071399150,1cyclictest3947-21diskmemload10:30:101
1071399150,14cyclictest0-21swapper/111:18:521
1070899154,2cyclictest141rcu_preempt12:20:170
1070899151,7cyclictest30197-21munin-run11:40:000
10708991511,4cyclictest19641-21apache210:39:270
1070899150,1cyclictest0-21swapper/009:17:110
1071999141,13cyclictest0-21swapper/309:15:223
10719991410,2cyclictest29152-21sed11:00:153
10717991413,1cyclictest0-21swapper/210:38:212
10717991413,1cyclictest0-21swapper/209:25:162
10717991411,2cyclictest18286-21sed12:10:252
1071799140,12cyclictest0-21swapper/210:40:152
1071399149,3cyclictest31978-21ssh11:40:191
10713991411,2cyclictest26014-21ntpq12:15:201
1071399140,14cyclictest0-21swapper/112:25:041
1071399140,13cyclictest0-21swapper/109:10:211
1070899147,6cyclictest24249-21munin-run12:15:000
1071999139,3cyclictest19201-21cron08:35:003
1071999139,3cyclictest11619-21sed08:15:263
1071999138,2cyclictest0-21swapper/312:38:193
10719991312,1cyclictest0-21swapper/311:56:143
10719991312,1cyclictest0-21swapper/310:00:333
10719991310,2cyclictest3545-21sed09:05:273
10719991310,2cyclictest28952-21ssh09:24:443
10719991310,2cyclictest19718-21chrt10:15:513
1071999130,1cyclictest0-21swapper/309:56:333
1071999130,13cyclictest0-21swapper/311:19:263
1071999130,12cyclictest31584-21cron09:00:003
1071999130,11cyclictest12176-21tr07:10:213
1071799139,3cyclictest31156-21grep08:55:282
10717991310,2cyclictest9792-21wc08:15:002
10717991310,2cyclictest3947-21diskmemload11:40:152
10717991310,2cyclictest31862-21grep09:45:162
10717991310,2cyclictest31579-21grep11:20:262
10717991310,2cyclictest29116-21grep11:00:152
10717991310,2cyclictest23893-21tail07:35:182
10717991310,2cyclictest16644-21grep07:20:182
10717991310,2cyclictest11936-21cat07:10:192
1071799130,13cyclictest0-21swapper/212:04:362
1071799130,13cyclictest0-21swapper/211:57:332
1071799130,12cyclictest2933-21grep09:05:202
1071799130,12cyclictest18721-21grep08:30:242
1071399136,5cyclictest38150irq/131-snd_hda10:45:231
10713991310,2cyclictest4092-21wc09:10:001
10713991310,2cyclictest31610-21systemctl09:00:001
10713991310,2cyclictest29136-21cat08:55:001
10713991310,2cyclictest11088-21grep07:10:151
1071399130,9cyclictest15705-21ssh11:50:161
1071399130,7cyclictest0-21swapper/111:56:251
1070899139,3cyclictest19756-21tr08:35:170
1070899137,5cyclictest19617-21apache209:11:350
1070899133,3cyclictest17476-21apache_volume12:30:100
1070899132,5cyclictest12320-21/usr/sbin/munin08:20:120
1070899131,6cyclictest30545-21perf09:45:000
1070899131,5cyclictest49-21kcompactd008:17:230
10708991310,2cyclictest924-21wc12:00:250
10708991310,2cyclictest6701-21grep11:45:140
10708991310,2cyclictest5309-21basename09:30:000
10708991310,2cyclictest25988-21tr07:40:150
10708991310,2cyclictest25825-21wc12:15:190
10708991310,2cyclictest2130-21grep07:55:270
1070899130,7cyclictest2461-21idleruntime-cro10:45:000
1070899130,2cyclictest141rcu_preempt10:58:440
1070899130,12cyclictest5622-21grep11:25:120
1070899130,12cyclictest18330-21grep08:30:200
1070899130,12cyclictest11082-21grep12:25:260
1071999129,2cyclictest8191-21apt10:10:003
1071999129,2cyclictest7447-21head11:45:183
1071999129,2cyclictest32149-21tr09:00:123
1071999129,2cyclictest32149-21tr09:00:123
1071999129,2cyclictest27732-21ssh10:59:573
1071999129,2cyclictest24657-21tr08:45:143
1071999129,2cyclictest17768-21tr12:30:123
1071999129,2cyclictest16833-21ls08:30:003
1071999129,2cyclictest15200-21grep07:15:283
1071999129,2cyclictest10334-21grep10:30:143
1071999128,2cyclictest22674-21grep11:35:153
1071999127,4cyclictest10658-21sh11:27:493
1071999124,4cyclictest15415-21grep11:30:233
1071999121,8cyclictest7580-21apt-get08:10:153
10719991212,0cyclictest0-21swapper/310:20:023
1071799129,2cyclictest4329-21rm11:05:132
1071799129,2cyclictest27151-21tr08:50:132
1071799129,2cyclictest22334-21ssh11:35:122
1071799129,2cyclictest21268-21ssh11:53:442
1071799129,2cyclictest16862-21cat08:30:002
1071799129,2cyclictest15657-21turbostat.cron07:20:002
1071799129,2cyclictest14336-21tr11:30:152
1071799129,2cyclictest10482-21ssh10:49:432
1071799127,3cyclictest18727-21cut12:30:172
1071799125,3cyclictest225-21systemd-journal07:55:002
1071799121,10cyclictest0-21swapper/209:42:402
1071799120,1cyclictest0-21swapper/210:18:122
1071799120,10cyclictest0-21swapper/211:26:012
1071399129,2cyclictest7741-21grep09:30:271
1071399129,2cyclictest32092-21sed07:50:251
1071399129,2cyclictest27994-21head12:35:281
1071399129,2cyclictest14922-21cut07:15:241
1071399129,2cyclictest12820-21grep10:50:231
1071399129,2cyclictest11095-21tr08:15:201
1071399127,4cyclictest24119-21ssh10:38:151
1071399127,4cyclictest19533-21awk07:25:191
1071399126,2cyclictest225-21systemd-journal11:25:001
10713991211,1cyclictest0-21swapper/110:01:441
1071399120,8cyclictest225-21systemd-journal07:50:001
1071399120,12cyclictest0-21swapper/109:38:311
1070899129,2cyclictest8773-21tr12:05:160
1070899129,2cyclictest6772-21wc08:05:260
1070899129,2cyclictest4339-21ssh11:05:130
1070899129,2cyclictest30588-21head08:55:220
1070899129,2cyclictest29300-21sed07:45:200
1070899129,2cyclictest26481-21grep10:20:190
1070899129,2cyclictest22140-21basename09:40:000
1070899129,2cyclictest21026-21tr07:30:160
1070899129,2cyclictest13786-21tr07:15:160
1070899129,2cyclictest11909-21wc07:10:190
1070899129,2cyclictest11025-21ls10:50:010
1070899125,6cyclictest3807-21munin-run11:05:000
1070899125,2cyclictest225-21systemd-journal08:00:190
10708991210,1cyclictest25610-21cut08:45:200
1070899120,11cyclictest1562-21cron09:05:000
1070899120,11cyclictest1562-21cron09:05:000
1071999118,2cyclictest9232-21ssh09:12:343
1071999118,2cyclictest8922-21ssh09:31:183
1071999118,2cyclictest8019-21basename09:50:183
1071999118,2cyclictest6181-21latency08:05:193
1071999118,2cyclictest4582-21cat11:05:153
1071999118,2cyclictest3667-21ssh11:23:563
1071999118,2cyclictest32191-21sh09:45:193
1071999118,2cyclictest31642-21awk07:50:203
1071999118,2cyclictest31108-21expr11:40:143
1071999118,2cyclictest28399-21ssh12:16:443
1071999118,2cyclictest26520-21head07:40:183
1071999118,2cyclictest25101-21ssh09:40:463
1071999118,2cyclictest22224-21cat08:40:123
1071999118,2cyclictest21629-21cron08:40:003
1071999118,2cyclictest20984-21tr07:30:163
1071999118,2cyclictest20808-21ssh10:35:403
1071999118,2cyclictest2059-21grep07:55:263
1071999118,2cyclictest1789-21ssh12:00:463
1071999118,2cyclictest17439-21ssh09:36:253
1071999118,2cyclictest16106-21ssh10:52:323
1071999118,2cyclictest15387-21taskset11:10:583
1071999118,2cyclictest12945-21ssh12:26:513
1071999118,2cyclictest11148-21ssh12:06:213
1071999117,3cyclictest9358-21cut10:10:163
1071999116,2cyclictest2689-21ssh10:45:023
1071999112,8cyclictest29773-21ssh10:41:053
1071799119,1cyclictest5962-21sed08:05:172
1071799118,2cyclictest9452-21ssh11:46:052
1071799118,2cyclictest32342-21ssh10:24:062
1071799118,2cyclictest24799-21grep08:45:162
1071799118,2cyclictest2060-21expr10:25:162
1071799118,2cyclictest19899-21expr08:35:162
1071799118,2cyclictest16082-21ssh11:11:302
1071799118,2cyclictest1146-21grep07:55:172
1071799118,2cyclictest11141-21wc10:30:192
1071799116,4cyclictest0-21swapper/210:00:032
1071799112,8cyclictest10392-21/usr/sbin/munin12:25:202
1071399118,2cyclictest9467-21ssh10:10:161
1071399118,2cyclictest636-21ssh10:43:381
1071399118,2cyclictest5504-21ssh10:26:541
1071399118,2cyclictest2930-21tr08:00:131
1071399118,2cyclictest26438-21grep07:40:181
1071399118,2cyclictest24675-21ssh11:35:401
1071399118,2cyclictest23309-21egrep07:35:161
1071399118,2cyclictest22654-21cut09:40:111
1071399118,2cyclictest20574-21grep10:55:181
1071399118,2cyclictest20414-21ssh09:19:381
1071399118,2cyclictest16837-21ssh11:12:051
1071399118,2cyclictest16696-21grep07:20:181
1071399118,2cyclictest16462-21ssh12:10:131
1071399118,2cyclictest10041-21ssh11:46:321
1071399117,3cyclictest9826-21grep12:05:261
1071399117,3cyclictest23574-21sed08:40:241
1071399116,3cyclictest14949-21tr08:25:141
1071399113,4cyclictest225-21systemd-journal09:03:261
1071399113,4cyclictest225-21systemd-journal09:03:261
1071399113,4cyclictest225-21systemd-journal08:00:001
1071399111,2cyclictest225-21systemd-journal10:20:001
1071399111,2cyclictest225-21systemd-journal08:35:001
1071399110,10cyclictest7680-21sed09:50:151
1070899119,1cyclictest28128-21tr08:50:200
1070899119,1cyclictest26244-21awk12:35:160
1070899118,2cyclictest30708-21tr07:50:130
1070899118,2cyclictest29871-21tr11:20:140
1070899118,2cyclictest23341-21tail09:20:260
1070899118,2cyclictest23168-21grep11:15:280
1070899118,2cyclictest1924-21grep10:25:160
1070899118,2cyclictest15143-21ssh09:55:000
1070899117,3cyclictest15449-21wc11:30:230
1070899116,3cyclictest23376-21/usr/sbin/munin11:55:220
1070899110,2cyclictest141rcu_preempt10:31:120
1070899110,0cyclictest0-21swapper/010:06:170
1071999108,1cyclictest27095-21cat08:50:133
1071999107,2cyclictest29769-21sed07:45:253
1071999107,2cyclictest23574-21expr07:35:153
1071999107,2cyclictest16543-21sort07:20:193
1071999107,2cyclictest13135-21grep08:20:173
1071999100,9cyclictest4691-21chrt08:02:003
1071999100,9cyclictest18953-21grep07:25:143
1071799108,2cyclictest7429-21sh09:30:252
1071799108,2cyclictest21998-21grep11:15:182
1071799108,1cyclictest6609-21ssh09:10:332
1071799108,1cyclictest22343-21expr08:40:122
1071799108,1cyclictest12292sleep209:00:532
1071799108,1cyclictest12292sleep209:00:532
1071799107,2cyclictest3065-21tr08:00:142
1071799107,2cyclictest26506-21head07:40:182
1071799107,2cyclictest21297-21grep07:30:172
1071799107,2cyclictest19055-21grep07:25:152
1071799105,3cyclictest0-21swapper/210:10:122
1071799104,2cyclictest0-21swapper/207:45:112
1071799100,9cyclictest12875-21ssh09:15:132
1071399108,1cyclictest26534-21seq08:48:571
1071399107,2cyclictest6103-21cat08:05:181
1071399107,2cyclictest20120-21entropy08:35:171
1071399100,9cyclictest13001-21grep08:20:161
1070899108,1cyclictest9390-21chrt08:11:220
1070899108,1cyclictest87052sleep009:31:080
1070899108,1cyclictest31357-21diskstats11:40:150
1070899108,1cyclictest2903-21sed09:05:200
1070899108,1cyclictest18408-21hostname07:25:110
1070899108,1cyclictest13472-21chrt10:50:300
1070899107,2cyclictest8848-21cut10:10:140
1070899107,2cyclictest32292-21cut09:45:190
1070899107,2cyclictest23824-21grep07:35:180
1070899107,2cyclictest23823-21taskset08:42:250
1070899107,2cyclictest15874-21grep08:25:200
107179997,1cyclictest12563-21cat08:20:142
107179997,1cyclictest10291-21expr08:15:142
107089997,1cyclictest15991-21hostname07:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional