You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 04:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Mar 15, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
255429859,35sleep10-21swapper/119:06:341
267328775,8sleep00-21swapper/019:08:060
197828758,27sleep20-21swapper/219:05:222
273828457,9sleep30-21swapper/319:08:573
2946992423,1cyclictest0-21swapper/321:46:173
2939992222,0cyclictest0-21swapper/121:38:491
2939992220,1cyclictest19261-21apache_accesses20:50:121
293999220,16cyclictest0-21swapper/123:56:371
294199210,18cyclictest0-21swapper/200:27:142
294699200,2cyclictest0-21swapper/323:54:103
294199200,1cyclictest0-21swapper/200:22:092
293999200,0cyclictest0-21swapper/100:00:091
294699192,13cyclictest0-21swapper/322:47:303
2946991918,1cyclictest0-21swapper/322:14:583
294699190,2cyclictest0-21swapper/322:57:073
294199190,19cyclictest0-21swapper/200:19:532
2939991918,1cyclictest38150irq/131-snd_hda00:10:131
293699196,6cyclictest14240-21idleruntime-cro00:15:000
2936991916,2cyclictest17241-21idleruntime-cro22:40:000
293699190,1cyclictest0-21swapper/022:43:430
294699180,18cyclictest397-21kworker/3:219:27:273
2936991817,1cyclictest0-21swapper/023:59:220
2946991715,2cyclictest39-21ksoftirqd/323:40:143
2946991715,1cyclictest0-21swapper/322:24:513
294199172,8cyclictest30745-21df_abs00:05:152
293999171,1cyclictest28516-21diskmemload23:21:261
293699170,1cyclictest0-21swapper/023:09:470
293699170,17cyclictest0-21swapper/021:38:130
294199167,5cyclictest19391-21/usr/sbin/munin23:00:162
294199165,1cyclictest141rcu_preempt00:31:302
294199160,0cyclictest0-21swapper/221:17:292
293999168,6cyclictest11731-21grep19:25:221
293999163,6cyclictest38150irq/131-snd_hda21:25:201
2939991614,1cyclictest0-21swapper/123:16:031
2939991612,2cyclictest7155-21sh21:15:341
293999160,16cyclictest0-21swapper/123:14:121
293699164,7cyclictest11836-21awk20:35:000
293699162,7cyclictest23768-21munin-run21:00:000
293699160,12cyclictest0-21swapper/021:30:120
2946991514,1cyclictest0-21swapper/300:24:523
2946991511,3cyclictest9610-21tr22:15:233
2941991514,1cyclictest0-21swapper/223:17:532
294199151,13cyclictest25685-21sed21:00:262
2941991511,3cyclictest0-21swapper/221:42:432
294199151,13cyclictest0-21swapper/220:00:122
2941991510,4cyclictest30117-21wc21:10:192
294199150,2cyclictest0-21swapper/200:03:512
294199150,1cyclictest0-21swapper/221:25:142
294199150,15cyclictest0-21swapper/223:44:292
293999159,5cyclictest20468-21rm23:40:001
2939991513,1cyclictest0-21swapper/122:57:031
293999150,3cyclictest32355-21rm23:46:211
293999150,13cyclictest17585-21sed22:20:161
2936991511,3cyclictest27502-21grep20:00:000
2946991413,1cyclictest0-21swapper/323:06:323
2946991411,2cyclictest512-21cron22:30:003
2946991410,3cyclictest15348-21cat00:35:003
294699140,1cyclictest0-21swapper/323:37:563
294199145,5cyclictest418-21kworker/u8:023:50:282
294199142,4cyclictest0-21swapper/220:50:132
2941991414,0cyclictest0-21swapper/223:37:062
2941991410,3cyclictest14840-21cut23:55:212
293999144,8cyclictest13244-21/usr/sbin/munin21:20:141
2939991413,1cyclictest0-21swapper/121:14:581
2939991412,1cyclictest0-21swapper/121:55:131
2939991412,1cyclictest0-21swapper/100:15:141
2939991410,3cyclictest13316-21ssh22:36:561
2939991410,2cyclictest31904-21cat00:05:251
293699143,4cyclictest7298-21/usr/sbin/munin00:30:250
293699141,1cyclictest0-21swapper/000:20:140
2936991411,2cyclictest3828-21wc20:15:220
2936991411,2cyclictest21392-21basename20:55:000
293699140,14cyclictest0-21swapper/021:40:410
294699131,5cyclictest0-21swapper/300:15:143
2946991313,0cyclictest0-21swapper/300:11:003
2946991310,2cyclictest5268-21cut21:15:143
2946991310,2cyclictest32451-21uname21:50:203
2946991310,2cyclictest14686-21cut21:20:193
294699130,1cyclictest0-21swapper/323:10:263
294699130,12cyclictest31951-21sed21:30:253
294199139,3cyclictest300-21df22:10:142
294199139,3cyclictest10235-21ls19:24:592
294199139,2cyclictest18168-21basename23:00:002
294199131,9cyclictest0-21swapper/200:10:132
2941991310,2cyclictest9098-21grep21:55:262
2941991310,2cyclictest5281-21wc23:30:212
2941991310,2cyclictest4622-21basename23:50:002
2941991310,2cyclictest2477-21wc22:30:202
2941991310,2cyclictest24440-21cat21:45:252
2941991310,2cyclictest18953-21grep19:40:192
2941991310,2cyclictest14740-21grep20:40:142
2941991310,2cyclictest14740-21grep20:40:142
294199130,13cyclictest0-21swapper/222:49:312
293999139,2cyclictest3334-21ssh22:50:221
293999139,2cyclictest30436-21apt-get21:50:001
293999139,2cyclictest30409-21tr20:05:131
293999135,2cyclictest225-21systemd-journal20:30:001
2939991310,2cyclictest8009-21cat19:20:011
2939991310,2cyclictest2439-21cat20:15:001
293699139,3cyclictest20040-21rm23:00:210
293699139,2cyclictest14104-21grep19:30:220
293699131,6cyclictest9459-21munin-run20:30:000
2936991310,2cyclictest4136-21head19:10:170
2936991310,2cyclictest3307-21tr23:10:160
2936991310,2cyclictest21353-21tr19:45:200
2936991310,2cyclictest11103-21tail22:55:140
293699130,2cyclictest0-21swapper/000:05:150
293699130,1cyclictest0-21swapper/022:50:140
293699130,11cyclictest0-21swapper/023:28:350
294699129,2cyclictest8357-21cut19:20:143
294699129,2cyclictest7828-21grep21:35:263
294699129,2cyclictest7441-21grep19:15:273
294699129,2cyclictest6020-21sed23:30:273
294699129,2cyclictest3386-21grep19:10:143
294699129,2cyclictest32638-21grep20:10:133
294699129,2cyclictest2943-21grep22:30:253
294699129,2cyclictest29309-21ssh21:10:143
294699129,2cyclictest15102-21ssh23:16:543
294699127,3cyclictest21707-21apt-get21:25:143
294699126,5cyclictest31292-21apache223:28:573
294699125,4cyclictest0-21swapper/300:25:163
294699123,7cyclictest0-21swapper/300:09:553
294699120,12cyclictest0-21swapper/322:52:473
294699120,11cyclictest26548-21cut21:05:133
294699120,0cyclictest0-21swapper/321:59:403
294199129,2cyclictest8528-21grep22:15:162
294199129,2cyclictest30823-21sed21:30:162
294199129,2cyclictest28508-21cron21:10:002
294199129,2cyclictest24142-21tr22:05:132
294199129,2cyclictest21480-21sed23:20:252
294199129,2cyclictest19232-21mailstats22:40:202
294199129,2cyclictest12324-21tr20:35:142
294199129,2cyclictest10028-21cat22:55:012
294199128,3cyclictest729-21grep21:50:262
294199128,3cyclictest30921-21head20:05:152
294199128,3cyclictest16624-21ssh00:35:172
294199121,7cyclictest6538-21df_abs21:35:152
2941991210,2cyclictest32-21ksoftirqd/222:35:152
294199120,10cyclictest0-21swapper/221:22:152
293999129,2cyclictest7933-21chrt22:15:031
293999129,2cyclictest758-21sed21:50:261
293999129,2cyclictest5946-21tail23:50:171
293999129,2cyclictest29002-21grep23:05:261
293999129,2cyclictest26937-21sed22:45:171
293999129,2cyclictest25740-21grep21:00:261
293999129,2cyclictest23276-21grep20:55:251
293999129,2cyclictest21055-21tr23:40:141
293999129,2cyclictest20960-21grep19:45:171
293999129,2cyclictest19600-21grep22:40:231
293999129,2cyclictest13300-21ls20:35:211
293999129,2cyclictest11423-21sed20:30:271
293999128,3cyclictest31864-21wc22:10:001
293999128,3cyclictest18548-21head20:45:251
293999127,4cyclictest24356-21ssh00:20:211
293999127,3cyclictest537-21snmpd19:35:021
293999125,4cyclictest31405-21apt-get00:25:131
2939991210,1cyclictest7252sleep123:27:561
293999120,12cyclictest0-21swapper/122:03:101
293699129,2cyclictest7958-21grep21:55:170
293699129,2cyclictest4925-21sed23:30:170
293699129,2cyclictest32450-21uname00:25:190
293699129,2cyclictest32349-21perf20:10:000
293699129,2cyclictest28013-21grep21:05:250
293699129,2cyclictest19299-21taskset23:20:000
293699129,2cyclictest16049-21ssh22:00:150
293699129,2cyclictest13919-21ssh21:20:130
293699129,2cyclictest12852-21sed20:35:170
293699127,4cyclictest20285-21munin-run23:40:000
293699121,6cyclictest543-21awk22:30:000
2936991212,0cyclictest0-21swapper/023:52:310
294699118,2cyclictest9192-21taskset22:35:053
294699118,2cyclictest6610-21sed20:20:243
294699118,2cyclictest30681-21ssh23:45:223
294699118,2cyclictest25185-21apache_processe19:55:113
294699118,2cyclictest25084-21ssh22:05:193
294699118,2cyclictest24428-21ssh00:00:533
294699118,2cyclictest20794-21tr23:20:193
294699118,2cyclictest18984-21expr23:00:143
294699118,2cyclictest18361-21ssh22:40:133
294699118,2cyclictest18083-21tr20:45:203
294699118,2cyclictest17985-21ssh00:35:393
294699118,2cyclictest16149-21cut22:00:163
294699118,2cyclictest15707-21ssh23:55:373
294699118,2cyclictest15204-21sort21:40:173
294699118,2cyclictest12919-21cut19:30:143
294699118,2cyclictest11626-21taskset20:32:403
294699117,3cyclictest19558-21sed19:40:253
294699117,3cyclictest17608-21sendmail-msp19:40:003
294699110,10cyclictest0-21swapper/320:15:143
294199118,2cyclictest911-21ssh23:28:052
294199118,2cyclictest6814-21taskset20:21:502
294199118,2cyclictest6116-21ssh23:11:192
294199118,2cyclictest4094-21cat20:15:262
294199118,2cyclictest22236-21grep20:55:162
294199118,2cyclictest20087-21ssh22:02:282
294199118,2cyclictest12349-21chrt19:26:502
294199118,2cyclictest10822-21uname20:30:212
294199117,3cyclictest17010-21grep22:20:132
294199113,4cyclictest225-21systemd-journal20:45:172
294199111,5cyclictest27291-21cron20:00:002
294199110,11cyclictest0-21swapper/222:25:272
293999119,2cyclictest0-21swapper/123:01:571
293999119,1cyclictest12932-21cut19:30:141
293999118,2cyclictest9494-21tr19:20:211
293999118,2cyclictest8637-21tr00:30:201
293999118,2cyclictest5244-21ssh22:32:091
293999118,2cyclictest24964-21ssh22:25:101
293999118,2cyclictest23368-21grep19:50:151
293999118,2cyclictest2258-21ssh22:10:461
293999118,2cyclictest1671-21ssh21:32:031
293999118,1cyclictest27725-21egrep20:00:131
293999117,3cyclictest5450-21cron19:15:011
293999117,3cyclictest26945-21taskset19:55:371
293999111,4cyclictest225-21systemd-journal20:20:111
2939991111,0cyclictest0-21swapper/100:36:061
293999110,10cyclictest4578-21cat23:30:151
293999110,10cyclictest15636-21ssh21:40:201
293699119,2cyclictest13-21ksoftirqd/000:18:200
293699118,2cyclictest9359-21ssh21:17:180
293699118,2cyclictest882-21head20:10:160
293699118,2cyclictest7652-21taskset19:17:420
293699118,2cyclictest7034-21taskset20:24:390
293699118,2cyclictest4452-21ssh21:14:580
293699118,2cyclictest431-21awk22:10:150
293699118,2cyclictest28209-21ssh21:48:150
293699118,2cyclictest26102-21ssh22:45:120
293699118,2cyclictest24219-21grep19:50:240
293699118,2cyclictest24055-21ssh21:25:440
293699118,2cyclictest23815-21ssh23:22:110
293699118,2cyclictest22990-21wc00:00:200
293699118,2cyclictest22449-21ssh00:39:100
293699118,2cyclictest20955-21grep23:40:140
293699118,2cyclictest1859-21head22:30:160
293699118,2cyclictest16591-21tr19:35:190
293699118,2cyclictest16547-21ssh22:20:040
293699117,3cyclictest8450-21ssh22:15:160
293699115,2cyclictest225-21systemd-journal20:44:550
293699115,2cyclictest225-21systemd-journal20:44:550
293699110,10cyclictest32694-21gzip21:50:220
294699108,1cyclictest31599-21tr20:05:213
294699108,1cyclictest244182sleep319:51:233
294699108,1cyclictest23994-21taskset21:00:053
294699108,1cyclictest21666-21hostname20:55:133
294699108,1cyclictest16758-21cat20:45:003
294699108,1cyclictest16758-21cat20:45:003
294699108,1cyclictest12912-21basename20:35:173
294699107,2cyclictest27929-21expr20:00:143
294699100,9cyclictest72942sleep320:25:013
294199108,1cyclictest7320-21basename20:25:112
294199108,1cyclictest29337-21ssh23:05:402
294199108,1cyclictest22808-21cat19:50:132
294199107,2cyclictest6676-21cut19:15:202
294199107,2cyclictest4133-21sed19:10:172
294199107,2cyclictest22048-21taskset19:46:272
294199100,9cyclictest1602-21cat20:10:252
294199100,9cyclictest0-21swapper/219:30:152
293999108,1cyclictest3703-21ntpq20:15:211
293999108,1cyclictest17830-21cat19:40:111
293999108,1cyclictest16412-21taskset20:42:551
293999108,1cyclictest16412-21taskset20:42:551
293999107,2cyclictest28451-21chrt21:09:191
293699108,1cyclictest26144-21ssh22:05:390
293699108,1cyclictest24904-21cat21:00:180
293699108,1cyclictest196862sleep019:41:350
293699108,1cyclictest10882-21expr19:25:150
293699107,2cyclictest29070-21ssh23:45:110
293699107,2cyclictest27728-21wc20:00:130
293699107,2cyclictest17149-21cat20:45:140
29469997,1cyclictest212292sleep320:53:183
29469997,1cyclictest20297-21wget19:45:143
29419991,1cyclictest225-21systemd-journal19:35:022
29369997,1cyclictest9211-21uname19:20:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional