You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 09:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 09, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19049210491,9sleep20-21swapper/219:05:552
1723729381,8sleep00-21swapper/019:05:090
1806928571,9sleep10-21swapper/119:05:181
1891128067,9sleep30-21swapper/319:05:293
1950899261,21cyclictest0-21swapper/121:35:101
19514992521,2cyclictest39-21ksoftirqd/322:10:473
1950899250,3cyclictest0-21swapper/122:48:171
1950899240,23cyclictest0-21swapper/100:37:441
1950899240,1cyclictest0-21swapper/122:06:251
248992230,0sleep10-21swapper/122:15:031
19514992321,2cyclictest39-21ksoftirqd/321:37:173
123942230,1sleep125-21ksoftirqd/122:26:051
1951499221,6cyclictest39-21ksoftirqd/300:17:443
1951499221,20cyclictest0-21swapper/300:29:103
1951499220,1cyclictest0-21swapper/322:35:133
1950899223,2cyclictest11106-21sed21:05:201
19508992212,6cyclictest38150irq/131-snd_hda21:50:131
1950899221,20cyclictest0-21swapper/121:55:131
1951499212,3cyclictest381rcuc/322:30:223
1951499210,5cyclictest39-21ksoftirqd/323:01:243
1950499211,1cyclictest0-21swapper/022:06:170
1951499207,12cyclictest8749-21tail00:00:163
1951499204,6cyclictest141rcu_preempt22:54:273
19514992019,1cyclictest12035-21diskmemload21:51:543
1951499200,19cyclictest0-21swapper/300:24:423
1950899200,2cyclictest0-21swapper/123:09:491
1950899200,0cyclictest0-21swapper/122:59:431
1950499207,6cyclictest4995-21munin-run21:25:000
19514991916,2cyclictest0-21swapper/300:35:173
1951499190,2cyclictest0-21swapper/322:42:233
1951099198,5cyclictest12185-21apt-get23:05:122
1951099190,18cyclictest0-21swapper/223:33:052
1951099190,18cyclictest0-21swapper/222:22:042
19508991917,1cyclictest30411-21/usr/sbin/munin20:40:211
19508991917,1cyclictest0-21swapper/120:55:131
1950899190,1cyclictest0-21swapper/123:57:081
1950499190,18cyclictest0-21swapper/023:00:150
19514991817,1cyclictest0-21swapper/321:29:063
1950899180,1cyclictest0-21swapper/121:18:101
94882170,1sleep10-21swapper/100:00:231
1951499175,5cyclictest39-21ksoftirqd/322:05:453
19514991716,1cyclictest39-21ksoftirqd/321:49:343
19514991715,1cyclictest26471-21apt-get00:30:143
1951099170,15cyclictest0-21swapper/223:35:292
1951099170,15cyclictest0-21swapper/200:27:332
19508991712,3cyclictest6034-21sh23:40:001
1950899170,1cyclictest0-21swapper/121:10:261
1951499167,3cyclictest39-21ksoftirqd/323:52:453
1951499166,10cyclictest39-21ksoftirqd/300:13:263
1951499163,1cyclictest141rcu_preempt22:02:203
19514991615,1cyclictest0-21swapper/322:20:453
19514991614,1cyclictest0-21swapper/321:33:113
19510991613,2cyclictest0-21swapper/222:53:422
1951099160,2cyclictest0-21swapper/221:56:552
1951099160,16cyclictest0-21swapper/221:28:562
19508991614,2cyclictest12035-21diskmemload00:30:271
1950499164,6cyclictest12035-21diskmemload00:20:110
1950499162,9cyclictest25169-21wget22:15:130
19504991614,1cyclictest12035-21diskmemload21:42:350
19514991513,1cyclictest29004-21apt-get23:15:113
1951499150,2cyclictest0-21swapper/321:11:563
1951499150,14cyclictest0-21swapper/322:26:343
1951099158,7cyclictest55650irq/127-enp2s0-23:55:062
19508991514,1cyclictest0-21swapper/119:25:111
1950899150,1cyclictest0-21swapper/123:30:161
1950899150,15cyclictest0-21swapper/123:00:421
1950899150,0cyclictest0-21swapper/120:49:521
1950499151,8cyclictest0-21swapper/022:55:210
1950499151,7cyclictest27789-21munin-run20:35:000
1950499150,1cyclictest0-21swapper/022:29:490
1950499150,13cyclictest26998-21grep00:10:250
1951499148,5cyclictest381rcuc/320:45:143
1951499147,5cyclictest12035-21diskmemload23:37:253
1951499145,1cyclictest17095-21ssh23:45:293
19514991413,1cyclictest12035-21diskmemload21:56:203
19514991412,2cyclictest39-21ksoftirqd/322:18:403
19514991412,2cyclictest0-21swapper/300:00:003
1951499141,11cyclictest381rcuc/323:40:113
1951499141,11cyclictest381rcuc/323:40:113
1951499140,3cyclictest0-21swapper/322:57:093
1951499140,2cyclictest0-21swapper/321:22:153
1951099149,4cyclictest6703-21/usr/sbin/munin21:45:212
1951099146,7cyclictest4298-21sh22:02:202
1951099142,12cyclictest29529-21kworker/u8:122:57:052
19510991412,2cyclictest32-21ksoftirqd/200:02:462
19510991410,3cyclictest7241-21cron21:00:002
19510991410,3cyclictest5376-21tr23:20:152
1950899148,5cyclictest2691-21cut22:40:131
1950899146,4cyclictest12975-21munin-run23:25:001
1950899143,4cyclictest225-21systemd-journal00:17:371
1950899143,1cyclictest141rcu_preempt21:43:381
19508991410,2cyclictest13999-21grep21:30:131
1950899140,14cyclictest0-21swapper/122:38:471
1950899140,13cyclictest0-21swapper/122:12:111
1950499145,5cyclictest225-21systemd-journal20:15:000
1950499143,5cyclictest0-21swapper/000:39:070
19504991411,2cyclictest1748-21cut00:15:150
1950499140,12cyclictest26579-21tr22:35:140
45612130,1sleep232-21ksoftirqd/221:43:422
1951499138,1cyclictest0-21swapper/320:05:133
1951499136,6cyclictest11597-21grep20:00:133
1951499136,6cyclictest11597-21grep20:00:133
1951499136,1cyclictest27503-21taskset20:31:343
1951499133,1cyclictest381rcuc/321:42:273
19514991311,2cyclictest39-21ksoftirqd/323:26:313
19514991310,2cyclictest18210-21grep00:05:263
1951099139,3cyclictest24550-21grep19:20:132
1951099139,3cyclictest22456-21chrt22:32:192
1951099139,2cyclictest11354-21cat21:05:242
19510991310,2cyclictest30914-21grep19:30:242
19510991310,2cyclictest29281-21grep20:35:212
19510991310,2cyclictest28176-21wc22:35:252
19510991310,2cyclictest21494-21sed23:10:162
19510991310,2cyclictest2021-21sed20:45:252
19510991310,2cyclictest12843-21grep21:10:142
19510991310,2cyclictest11237-21grep22:45:202
1951099130,12cyclictest0-21swapper/223:48:382
1951099130,12cyclictest0-21swapper/221:23:052
1951099130,11cyclictest0-21swapper/223:41:132
1951099130,11cyclictest0-21swapper/223:41:132
1950899139,3cyclictest24763-21tr19:20:141
1950899139,3cyclictest18253-21cut00:25:151
1950899138,4cyclictest28978-21ssh23:53:071
1950899138,3cyclictest15195-21wc23:25:251
19508991312,1cyclictest0-21swapper/122:33:091
19508991311,1cyclictest0-21swapper/123:15:491
19508991310,2cyclictest8822-21ssh22:24:521
19508991310,2cyclictest647-21tail22:00:161
1950899130,1cyclictest0-21swapper/121:48:071
1950499139,3cyclictest20632-21cron20:20:000
1950499131,1cyclictest21106-21apt-get21:15:140
19504991310,2cyclictest24616-21ssh22:15:000
19504991310,2cyclictest23800-21grep19:15:250
1951499129,2cyclictest6367-21grep20:55:213
1951499129,2cyclictest30149-21cron20:39:593
1951499129,2cyclictest25480-21wc20:30:013
1951499129,2cyclictest24243-21turbostat.cron19:20:003
1951499129,2cyclictest11043-21cron20:00:003
1951499128,3cyclictest29805-21cron23:35:003
1951499122,6cyclictest39-21ksoftirqd/323:10:123
1951499122,5cyclictest225-21systemd-journal19:10:443
1951499121,4cyclictest0-21swapper/321:17:563
19514991210,2cyclictest39-21ksoftirqd/321:00:283
1951099129,2cyclictest9011-21tr00:20:012
1951099129,2cyclictest655-21cat00:15:002
1951099129,2cyclictest3513-21tr20:50:162
1951099129,2cyclictest23069-21wc20:25:012
1951099129,2cyclictest22666-21grep19:15:142
1951099129,2cyclictest21872-21grep21:15:182
1951099128,3cyclictest0-21swapper/223:25:242
1951099128,2cyclictest26288-21tune2fs20:30:152
1951099125,6cyclictest1318-21cron19:40:002
1951099124,5cyclictest225-21systemd-journal23:15:282
1951099122,9cyclictest489-21cron19:50:002
19510991211,1cyclictest0-21swapper/221:36:452
1951099120,1cyclictest0-21swapper/220:07:022
1951099120,12cyclictest0-21swapper/200:05:442
1950899129,2cyclictest8251-21head21:00:161
1950899129,2cyclictest5886-21cat21:25:141
1950899129,2cyclictest20137-21sed20:15:251
1950899129,2cyclictest10654-21grep19:55:251
1950899128,2cyclictest16331-21tr20:10:141
1950899120,12cyclictest0-21swapper/123:47:391
1950899120,11cyclictest14392-21sort20:05:161
1950499129,2cyclictest9033-21sh22:25:000
1950499129,2cyclictest5800-21grep21:25:140
1950499129,2cyclictest32666-21cat19:35:200
1950499129,2cyclictest31884-21cat20:40:240
1950499129,2cyclictest24672-21ls23:50:180
1950499129,2cyclictest23786-21sed23:30:260
1950499129,2cyclictest15983-21cut21:50:170
1950499129,2cyclictest14049-21sed23:25:140
1950499128,3cyclictest2841-21head19:40:200
1950499124,3cyclictest0-21swapper/023:05:130
19504991211,1cyclictest0-21swapper/019:55:120
1950499120,1cyclictest0-21swapper/021:35:290
1951499118,2cyclictest7310-21head19:50:153
1951499118,2cyclictest7279-21chrt23:20:373
1951499118,2cyclictest3875-21cut20:50:193
1951499118,2cyclictest32022-21expr19:35:153
1951499118,2cyclictest30773-21cat19:30:233
1951499118,2cyclictest19309-21head20:15:163
1951499118,2cyclictest16173-21hostname20:10:133
1951499118,2cyclictest11511-21unixbench-2d21:05:243
1951499118,2cyclictest11396-21expr22:45:203
19514991110,1cyclictest12198-21ssh23:05:133
1951099118,2cyclictest9220-21head21:00:262
1951099118,2cyclictest9206-21cut19:55:142
1951099118,2cyclictest6620-21hostname19:50:122
1951099118,2cyclictest5103-21ntpq23:00:232
1951099118,2cyclictest3568-21cat22:40:182
1951099118,2cyclictest28855-21ssh22:16:452
1951099118,2cyclictest25403-21grep23:50:252
1951099118,2cyclictest21391-21grep19:10:242
1951099118,2cyclictest20143-21grep20:15:252
1951099118,2cyclictest19622-21sh22:11:072
1951099118,2cyclictest16882-21sed21:50:252
1951099118,2cyclictest16854-21ssh00:24:332
1951099118,2cyclictest15034-21ssh22:09:082
1951099118,2cyclictest11823-21sh22:25:392
1951099117,3cyclictest32-21ksoftirqd/200:35:192
1951099117,3cyclictest14196-21grep21:30:132
1951099116,2cyclictest225-21systemd-journal20:15:002
19510991110,1cyclictest55550irq/126-enp2s0-00:32:102
1950899119,1cyclictest22128-21aten2_r7power_c19:15:131
1950899118,2cyclictest8342-21ssh23:40:231
1950899118,2cyclictest8342-21ssh23:40:231
1950899118,2cyclictest3304-21grep19:40:251
1950899118,2cyclictest29278-21tr20:35:211
1950899118,2cyclictest25846-21fw_packets00:10:151
1950899118,2cyclictest24669-21ssh23:12:001
1950899118,2cyclictest20929-21apache_volume20:20:121
1950899118,2cyclictest19300-21sh22:50:121
1950899118,2cyclictest12721-21ssh00:21:221
1950899118,2cyclictest11599-21tr20:00:141
1950899118,2cyclictest11599-21tr20:00:131
1950899118,1cyclictest31166-21ssh21:20:251
1950899110,10cyclictest16754-21expr00:05:141
1950499119,1cyclictest12573-21grep20:00:210
1950499119,1cyclictest12573-21grep20:00:210
1950499118,2cyclictest9289-21ssh23:41:000
1950499118,2cyclictest9289-21ssh23:41:000
1950499118,2cyclictest8716-21head00:00:160
1950499118,2cyclictest678-21sed22:00:160
1950499118,2cyclictest6556-21ssh21:45:050
1950499118,2cyclictest5344-21tr23:20:150
1950499118,2cyclictest31761-21ssh23:55:060
1950499118,2cyclictest30007-21taskset20:38:160
1950499118,2cyclictest2796-21ssh23:18:410
1950499118,2cyclictest2571-21ssh23:37:190
1950499118,2cyclictest24682-21tail21:55:170
1950499118,2cyclictest23791-21chrt23:11:200
1950499118,2cyclictest22268-21rm22:32:100
1950499118,2cyclictest21645-21ssh22:50:420
1950499118,2cyclictest20445-21grep19:10:150
1950499118,2cyclictest18126-21cut00:25:150
1950499118,2cyclictest15909-21ssh21:11:290
1950499118,2cyclictest14940-21ssh23:45:100
1950499118,2cyclictest14662-21grep21:30:150
1950499118,2cyclictest13018-21ssh22:45:340
1950499117,3cyclictest5094-21uname19:45:190
1950499117,2cyclictest23815-21systemd-cgroups00:09:310
1950499115,6cyclictest1436-21sendmail-mta19:21:020
1950499115,2cyclictest225-21systemd-journal19:35:000
1950499113,5cyclictest0-21swapper/000:30:210
1950499110,10cyclictest62512sleep020:55:200
1951499108,1cyclictest30629-21grep20:40:143
1951499108,1cyclictest27233-21expr19:25:153
1951499108,1cyclictest24910-21expr19:20:143
1951499108,1cyclictest21236-21cat20:20:143
1951499107,2cyclictest3059-21cat19:40:243
1951499100,8cyclictest0-21swapper/319:45:193
1951099108,1cyclictest30674-21expr20:40:142
1951099107,2cyclictest25050-21taskset20:25:312
1951099107,2cyclictest12695-21sendmail_mailtr20:00:212
1951099107,2cyclictest12695-21sendmail_mailtr20:00:212
1951099100,9cyclictest27497-21cut19:25:162
1950899108,1cyclictest29879-21cut19:30:151
1950899107,2cyclictest851-21tail19:35:261
1950899107,2cyclictest4628-21cut19:45:151
1950899107,2cyclictest25902-21tr20:30:131
1950899107,2cyclictest23794-21grep20:25:161
1950899107,2cyclictest20570-21basename19:10:161
1950899103,5cyclictest537-21snmpd20:50:121
1950499108,1cyclictest23346-21taskset20:25:140
1950499107,2cyclictest7000-21expr19:50:130
1950499107,2cyclictest3347-21taskset20:50:150
1950499107,2cyclictest3224-21ssh22:40:150
1950499107,2cyclictest21210-21grep20:20:140
1950499107,2cyclictest14260-21df20:05:150
1950499107,2cyclictest1108-21head20:45:150
195109996,2cyclictest1846-21tr19:40:132
195049997,1cyclictest7674-21wget21:00:140
195049997,1cyclictest10361-21expr21:05:130
195049996,2cyclictest28559-21grep19:25:260
195089986,1cyclictest8032-21sed19:50:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional