You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 21:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 13, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2418129481,9sleep20-21swapper/207:07:562
2432929366,9sleep30-21swapper/307:09:503
2408228872,12sleep00-21swapper/007:06:380
2424228669,13sleep10-21swapper/107:08:441
229142550,0sleep00-21swapper/011:09:400
2443499291,27cyclictest21657-21kworker/3:011:54:333
2443299280,27cyclictest0-21swapper/210:20:312
24434992521,3cyclictest21657-21kworker/3:009:57:233
24434992422,1cyclictest21657-21kworker/3:009:33:433
2442899240,22cyclictest0-21swapper/110:11:011
24434992321,2cyclictest21657-21kworker/3:010:27:533
24432992321,1cyclictest32-21ksoftirqd/209:55:002
24432992321,1cyclictest32-21ksoftirqd/209:55:002
24434992221,1cyclictest21657-21kworker/3:010:01:333
2443299220,21cyclictest0-21swapper/209:49:392
24428992219,2cyclictest27903-21python12:10:241
24426992221,1cyclictest0-21swapper/010:46:580
24434992119,2cyclictest21657-21kworker/3:012:37:533
24432992119,1cyclictest0-21swapper/212:27:162
2443299210,21cyclictest0-21swapper/212:35:512
24428992120,1cyclictest0-21swapper/111:36:111
24434992019,1cyclictest21657-21kworker/3:009:14:233
2443299203,13cyclictest29876-21python09:55:242
2443299200,2cyclictest0-21swapper/212:34:482
2443299200,2cyclictest0-21swapper/209:05:152
2442899200,1cyclictest0-21swapper/111:20:121
2442899200,1cyclictest0-21swapper/109:45:231
2442899200,11cyclictest0-21swapper/109:05:151
291512190,0sleep20-21swapper/211:13:032
24434991918,1cyclictest21657-21kworker/3:009:54:233
24434991918,1cyclictest21657-21kworker/3:009:54:233
24434991917,2cyclictest21657-21kworker/3:010:17:423
24434991917,2cyclictest21657-21kworker/3:009:35:533
2443299190,15cyclictest0-21swapper/209:27:342
2442899190,1cyclictest0-21swapper/110:06:181
24426991918,1cyclictest0-21swapper/009:49:400
58512180,0sleep00-21swapper/009:40:560
24434991817,1cyclictest21657-21kworker/3:009:48:233
24434991816,2cyclictest21657-21kworker/3:010:06:533
2443299180,17cyclictest0-21swapper/211:37:302
2443299180,17cyclictest0-21swapper/211:00:082
2442899180,2cyclictest0-21swapper/112:26:041
24434991715,2cyclictest21657-21kworker/3:012:34:133
24434991715,2cyclictest21657-21kworker/3:012:25:433
24434991715,2cyclictest21657-21kworker/3:010:36:333
24434991715,2cyclictest21657-21kworker/3:010:33:233
24434991715,2cyclictest21657-21kworker/3:009:21:533
2443299170,17cyclictest0-21swapper/209:17:252
24428991713,3cyclictest38150irq/131-snd_hda10:33:221
2442899171,15cyclictest23309-21ssh09:13:141
2442899170,1cyclictest0-21swapper/112:20:491
24426991713,1cyclictest2041-21/usr/sbin/munin09:20:180
2443499161,14cyclictest0-21swapper/312:12:573
24434991610,4cyclictest4453-21grep07:35:153
2443299161,7cyclictest0-21swapper/210:15:132
2442899162,13cyclictest17232-21diskmemload12:18:211
2442899160,15cyclictest0-21swapper/109:50:391
2442899160,15cyclictest0-21swapper/109:50:391
2442699161,2cyclictest141rcu_preempt11:57:030
2442699160,11cyclictest0-21swapper/008:15:130
22412160,1sleep10-21swapper/109:39:421
24434991511,2cyclictest12442-21cron08:59:593
2443499150,13cyclictest0-21swapper/309:26:533
2443499150,11cyclictest0-21swapper/311:25:133
24432991513,2cyclictest0-21swapper/209:40:002
24428991513,1cyclictest0-21swapper/112:03:561
24428991511,3cyclictest32088-21wc10:35:271
2442899151,12cyclictest2183-21aten2_r7power_e09:20:141
2442899150,1cyclictest0-21swapper/110:18:551
2442899150,15cyclictest0-21swapper/111:03:501
2442699151,14cyclictest0-21swapper/011:30:310
2442699150,8cyclictest8721-21idleruntime-cro07:44:590
2442699150,0cyclictest0-21swapper/012:34:480
2443499145,5cyclictest26903-21ssh09:15:163
24434991410,2cyclictest11080-21idleruntime-cro09:45:003
2443299148,5cyclictest32544-21apache211:17:272
2443299144,7cyclictest3044-21sed12:15:172
2443299140,14cyclictest0-21swapper/210:45:532
24428991413,1cyclictest0-21swapper/107:15:141
24428991410,2cyclictest14391-21grep07:55:141
2442899140,1cyclictest17232-21diskmemload11:49:161
2442699147,7cyclictest32544-21apache210:23:370
2442699144,5cyclictest17232-21diskmemload12:35:120
2443499139,3cyclictest7473-21munin-plugin-st11:20:003
2443499139,3cyclictest26345-21sed07:10:263
2443499131,1cyclictest32112-21ssh10:55:183
24434991310,2cyclictest9151-21sed11:00:263
24434991310,2cyclictest26851-21grep08:20:173
24434991310,2cyclictest14785-21grep10:45:153
24434991310,2cyclictest13451-21grep09:00:173
24434991310,2cyclictest1008-21ssh11:35:153
2443499130,2cyclictest0-21swapper/312:23:513
2443499130,1cyclictest0-21swapper/311:48:473
2443299138,2cyclictest6548-21munin-plugin-st11:00:002
2443299135,7cyclictest32544-21apache211:25:552
24432991310,3cyclictest0-21swapper/212:14:482
24432991310,2cyclictest4438-21cut10:00:142
24432991310,1cyclictest0-21swapper/211:52:352
2443299130,5cyclictest0-21swapper/211:09:332
2443299130,3cyclictest0-21swapper/209:41:492
2442899139,3cyclictest4352-21tr12:35:211
2442899139,2cyclictest1594-21sed08:35:171
24428991313,0cyclictest0-21swapper/112:31:531
24428991312,1cyclictest0-21swapper/111:15:391
24428991311,1cyclictest0-21swapper/109:17:271
24428991310,2cyclictest9730-21cut11:40:171
24428991310,2cyclictest7350-21sed07:40:181
24428991310,2cyclictest24953-21grep11:30:131
24428991310,2cyclictest21928-21cut08:10:161
24428991310,2cyclictest16232-21wc10:45:261
2442899130,2cyclictest0-21swapper/110:55:001
2442899130,13cyclictest0-21swapper/109:41:381
2442899130,12cyclictest6088-21cat10:00:261
2442899130,11cyclictest16729-21cpuspeed_turbos08:00:161
2442899130,11cyclictest16729-21cpuspeed_turbos08:00:161
24426991311,1cyclictest0-21swapper/011:00:040
24426991310,2cyclictest2932-21sh08:40:010
24426991310,2cyclictest23932-21sh11:30:000
2442699130,1cyclictest0-21swapper/009:05:150
2442699130,1cyclictest0-21swapper/008:00:150
2442699130,1cyclictest0-21swapper/008:00:150
2442699130,12cyclictest28021-21sed09:15:260
2442699130,11cyclictest0-21swapper/011:46:570
2443499129,2cyclictest3291-21grep08:40:163
2443499129,2cyclictest29986-21grep07:20:163
2443499129,2cyclictest25757-21grep11:10:263
2443499129,2cyclictest25458-21grep11:30:163
2443499129,2cyclictest24401-21grep08:15:163
2443499129,2cyclictest22070-21head08:10:163
2443499129,2cyclictest21829-21head10:10:173
2443499129,2cyclictest19588-21sort08:05:163
2443499124,4cyclictest225-21systemd-journal08:35:013
2443499121,8cyclictest5599-21apt-get08:45:143
2443499120,11cyclictest18324-21tr12:05:143
2443299129,2cyclictest26761-21cron07:15:002
2443299129,2cyclictest22956-21grep08:10:252
2443299129,2cyclictest13410-21sort09:00:162
2443299129,2cyclictest11691-21grep07:50:142
2443299124,4cyclictest13996-21munin-plugin-st10:45:002
2443299122,5cyclictest14145-21apt-get07:55:132
24432991210,1cyclictest11127-21cut08:55:182
2443299120,9cyclictest0-21swapper/211:20:132
2443299120,3cyclictest0-21swapper/211:47:352
2443299120,2cyclictest0-21swapper/210:11:472
2443299120,11cyclictest25841-21wc11:30:192
2443299120,11cyclictest200512sleep209:10:412
2443299120,11cyclictest16366-21sh12:23:282
2442899129,2cyclictest3328-21grep11:55:261
2442899129,2cyclictest25772-21cron08:20:001
2442899129,2cyclictest18181-21grep11:25:271
2442899129,2cyclictest11268-21grep09:25:171
2442899128,3cyclictest28572-21tr09:55:141
2442899128,2cyclictest10595-21grep08:55:151
2442899124,7cyclictest13549-21ssh10:44:381
2442899120,7cyclictest38150irq/131-snd_hda08:05:131
2442899120,12cyclictest17232-21diskmemload09:32:461
2442899120,0cyclictest0-21swapper/111:10:331
2442699129,2cyclictest9379-21cat08:50:230
2442699129,2cyclictest19605-21grep09:30:170
2442699129,2cyclictest16255-21idleruntime-cro08:00:000
2442699128,1cyclictest17232-21diskmemload09:11:470
2442699125,6cyclictest11076-21apt-get07:50:000
2442699125,4cyclictest17232-21diskmemload11:17:240
2442699122,2cyclictest29344-21sed09:55:180
2442699120,7cyclictest0-21swapper/009:54:590
2442699120,7cyclictest0-21swapper/009:54:590
2442699120,5cyclictest0-21swapper/010:59:120
2442699120,2cyclictest0-21swapper/012:18:370
165732120,1sleep20-21swapper/210:26:072
2443499118,2cyclictest8725-21sed08:50:153
2443499118,2cyclictest28758-21grep07:15:273
2443499118,2cyclictest22728-21ssh10:50:093
2443499118,2cyclictest1990-21sh12:15:033
2443499118,2cyclictest16492-21basename11:05:183
2443499118,2cyclictest13620-21ssh10:44:443
2443499118,2cyclictest1309-21ssh11:55:073
2443499118,2cyclictest12289-21rm10:24:173
2443499118,2cyclictest11705-21head12:00:273
2443499118,2cyclictest11645-21rm11:21:443
2443499117,2cyclictest17213-21grep08:00:183
2443499117,2cyclictest17213-21grep08:00:183
2443499116,2cyclictest3928-21sh07:35:003
2443299119,1cyclictest9062-21grep08:50:172
2443299119,1cyclictest29196-21sed08:25:172
2443299119,1cyclictest17321-21sed08:00:192
2443299119,1cyclictest17321-21sed08:00:192
2443299119,1cyclictest1722-21grep11:55:142
2443299118,2cyclictest9672-21acpi12:00:112
2443299118,2cyclictest518-21ssh10:35:532
2443299118,2cyclictest4156-21taskset07:35:042
2443299118,2cyclictest4059-21chrt08:40:202
2443299118,2cyclictest32642-21sed07:25:182
2443299118,2cyclictest24086-21ssh10:50:202
2443299118,2cyclictest14144-21wc10:05:232
2443299118,2cyclictest11739-21sh11:41:002
2443299114,1cyclictest18817-21apt-get09:30:132
2443299110,10cyclictest84122sleep207:41:262
2443299110,10cyclictest28011-21sed07:15:202
2443299110,10cyclictest24872-21rm10:31:062
2443299110,10cyclictest21607-21sh12:06:422
2443299110,0cyclictest0-21swapper/209:20:322
2442899119,1cyclictest4004-21grep08:40:191
2442899118,2cyclictest9184-21wc07:45:151
2442899118,2cyclictest4425-21ssh10:58:201
2442899118,2cyclictest31525-21grep08:30:161
2442899118,2cyclictest30124-21chrt11:52:291
2442899118,2cyclictest26614-21df08:20:161
2442899118,2cyclictest19406-21tr12:05:221
2442899118,2cyclictest15375-21ssh10:25:261
2442899118,2cyclictest11538-21cat07:50:111
2442899116,2cyclictest225-21systemd-journal10:25:001
2442899114,5cyclictest15560-21apt-get11:05:141
2442899111,6cyclictest225-21systemd-journal07:35:001
2442699119,1cyclictest32548-21grep08:30:260
2442699119,1cyclictest19658-21grep08:05:160
2442699118,2cyclictest9093-21aten2_r7power_e11:40:150
2442699118,2cyclictest5991-21taskset07:36:040
2442699118,2cyclictest5051-21grep10:00:170
2442699118,2cyclictest3516-21grep07:30:260
2442699118,2cyclictest29316-21ssh11:51:510
2442699118,2cyclictest29299-21ssh10:53:570
2442699118,2cyclictest26793-21expr12:10:140
2442699118,2cyclictest2669-21tail11:35:270
2442699118,2cyclictest22824-21grep10:30:160
2442699118,2cyclictest2108-21ssh10:37:080
2442699118,2cyclictest14048-21ssh12:02:160
2442699118,2cyclictest13427-21rm10:25:010
2442699117,3cyclictest7888-21ssh11:20:090
2442699117,3cyclictest537-21snmpd09:37:260
2442699117,3cyclictest13025-21cut09:00:140
2442699116,2cyclictest0-21swapper/010:42:330
2442699112,7cyclictest29439-21apache_volume07:20:130
2442699111,6cyclictest25087-21needreboot11:10:200
24426991110,1cyclictest0-21swapper/010:15:120
2443499108,1cyclictest9036-21wget07:45:153
2443499108,1cyclictest6553-21taskset07:40:053
2443499108,1cyclictest14254-21cut07:55:133
2443499108,1cyclictest127082sleep307:50:203
2443499108,1cyclictest11046-21ssh11:40:273
2443499107,2cyclictest29741-21cut08:25:223
2443499107,2cyclictest1552-21cron07:30:003
2443499100,9cyclictest15581-21cat09:05:163
2443299108,1cyclictest29569-21cut07:20:142
2443299108,1cyclictest25211-21wc08:15:252
2443299108,1cyclictest2300-21timerandwakeup08:35:252
2443299108,1cyclictest207102sleep208:06:102
2443299107,2cyclictest32400-21cat08:30:252
2443299107,2cyclictest26667-21cut08:20:172
2443299107,2cyclictest2317-21sed07:30:152
2443299107,2cyclictest10123-21ntp_states07:45:222
2442899108,1cyclictest13014-21expr09:00:141
2442899107,2cyclictest959-21cat07:25:261
2442899107,2cyclictest9219-21grep08:50:191
2442899107,2cyclictest5732-21wget08:45:151
2442899107,2cyclictest5293-21tr07:35:201
2442899107,2cyclictest29716-21tr07:20:151
2442899107,2cyclictest28475-21grep08:25:141
2442899106,3cyclictest26171-21wc07:10:261
2442699108,2cyclictest11763-21sh09:25:210
2442699108,1cyclictest5902-21cat08:45:150
2442699108,1cyclictest3333-21aten2_r7power_v08:40:160
2442699108,1cyclictest32620-21grep07:25:180
2442699108,1cyclictest28698-21grep08:25:150
2442699108,1cyclictest26035-21chrt08:20:100
2442699108,1cyclictest21067-21wget10:10:140
2442699108,1cyclictest11787-21ntp_kernel_pll_12:20:190
2442699106,1cyclictest225-21systemd-journal09:00:000
2442699102,3cyclictest21590-21sshd12:26:040
2442699100,9cyclictest12458-21ssh10:05:030
2442699100,1cyclictest0-21swapper/012:08:010
244349997,1cyclictest818-21apt-get08:35:133
244329997,1cyclictest7213-21grep08:45:252
244269998,1cyclictest25125-21expr07:10:150
244269998,1cyclictest13036-21expr07:50:250
244269997,2cyclictest28340-21chrt07:15:230
244269997,1cyclictest21996-21grep08:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional