You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 08:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Mar 14, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2937629279,9sleep00-21swapper/019:06:540
2958628560,9sleep30-21swapper/319:09:383
2956128460,13sleep10-21swapper/119:09:211
2929528371,8sleep20-21swapper/219:05:552
76252710,0sleep00-21swapper/023:35:320
2970499290,28cyclictest0-21swapper/122:10:231
2970099299,19cyclictest22437-21diskmemload21:25:000
2970099290,23cyclictest0-21swapper/019:45:120
2970099275,21cyclictest22437-21diskmemload22:16:140
2970099270,25cyclictest0-21swapper/023:15:100
2970699260,26cyclictest0-21swapper/200:14:392
2970099261,24cyclictest0-21swapper/022:50:190
2970499253,14cyclictest0-21swapper/100:25:131
2970099250,24cyclictest0-21swapper/000:26:130
2970099250,20cyclictest22437-21diskmemload22:58:320
2970699240,2cyclictest0-21swapper/200:34:002
2970099241,22cyclictest23864-21aten2_r7power_e21:30:130
2970099241,22cyclictest15473-21/usr/sbin/munin21:25:180
2970099241,22cyclictest11878-21apt-get23:00:110
2970099240,24cyclictest0-21swapper/021:39:030
2970099240,23cyclictest0-21swapper/023:51:380
2970099240,22cyclictest0-21swapper/000:39:150
2970099240,12cyclictest0-21swapper/019:15:120
2970699230,1cyclictest0-21swapper/222:02:402
2970099237,11cyclictest22437-21diskmemload23:23:190
2970099233,19cyclictest22437-21diskmemload23:08:060
2970099231,21cyclictest22437-21diskmemload22:09:180
2970099231,18cyclictest0-21swapper/019:55:140
2970099230,20cyclictest0-21swapper/022:26:110
29709992220,1cyclictest0-21swapper/300:21:093
29709992220,1cyclictest0-21swapper/300:21:093
2970999220,21cyclictest0-21swapper/321:21:073
29704992220,1cyclictest25-21ksoftirqd/123:54:211
2970499221,18cyclictest22437-21diskmemload22:41:281
2970099220,20cyclictest0-21swapper/021:44:170
2970099220,20cyclictest0-21swapper/020:40:150
119452220,0sleep20-21swapper/223:57:582
2970099216,14cyclictest22437-21diskmemload21:17:120
2970099213,15cyclictest2336-21latency_hist22:35:010
2970099212,18cyclictest22437-21diskmemload22:47:260
2970099211,19cyclictest22437-21diskmemload23:11:270
2970099211,17cyclictest22437-21diskmemload23:34:420
2970099210,20cyclictest7758-21sendmail-msp20:40:000
2970099210,19cyclictest0-21swapper/000:20:240
2970099210,19cyclictest0-21swapper/000:20:230
2970099210,17cyclictest22437-21diskmemload22:10:100
2970999200,19cyclictest0-21swapper/322:51:013
2970699200,19cyclictest0-21swapper/221:20:482
2970499200,1cyclictest22437-21diskmemload00:03:161
2970499200,19cyclictest0-21swapper/120:40:131
2970099206,12cyclictest13-21ksoftirqd/022:23:330
2970099203,16cyclictest22437-21diskmemload21:46:110
2970099202,16cyclictest0-21swapper/000:16:020
2970099201,18cyclictest22437-21diskmemload22:42:060
2970099201,18cyclictest22437-21diskmemload22:04:430
2970099201,16cyclictest8710-21unixbench_multi19:30:240
2970099200,17cyclictest0-21swapper/023:45:150
2970099200,10cyclictest0-21swapper/023:43:100
29709991919,0cyclictest0-21swapper/300:18:583
29706991919,0cyclictest0-21swapper/200:07:282
29706991918,1cyclictest0-21swapper/222:22:032
2970699190,18cyclictest0-21swapper/200:15:452
29704991918,1cyclictest0-21swapper/121:00:131
29704991914,1cyclictest0-21swapper/122:25:201
2970099191,15cyclictest879-21ssh21:55:110
2970099191,12cyclictest4666-21apt-get19:25:140
2970099190,18cyclictest0-21swapper/020:05:120
2970099190,16cyclictest0-21swapper/000:10:320
29709991815,2cyclictest0-21swapper/323:25:503
2970699184,6cyclictest0-21swapper/222:45:152
2970699180,0cyclictest0-21swapper/222:14:352
29704991816,1cyclictest25-21ksoftirqd/100:33:591
2970099186,11cyclictest13926-21munin-run19:45:000
2970099181,16cyclictest0-21swapper/000:00:190
2970099181,15cyclictest0-21swapper/020:20:120
2970099180,17cyclictest0-21swapper/022:36:250
2970099180,15cyclictest0-21swapper/023:55:310
2970099180,14cyclictest0-21swapper/000:05:050
2970999175,7cyclictest22236-21/usr/sbin/munin23:45:133
2970699170,17cyclictest0-21swapper/200:01:022
29704991717,0cyclictest0-21swapper/123:45:121
2970499171,1cyclictest141rcu_preempt00:05:511
2970499170,1cyclictest0-21swapper/123:21:091
2970499170,1cyclictest0-21swapper/122:51:201
2970099175,6cyclictest30819-21perf21:15:000
2970099172,14cyclictest22437-21diskmemload23:29:210
2970099171,15cyclictest1140-21ssh00:30:140
2970099171,14cyclictest25375-21expr21:50:140
2970999168,7cyclictest28297-21apache223:37:433
29706991615,1cyclictest32-21ksoftirqd/223:14:112
29706991614,2cyclictest32-21ksoftirqd/222:16:162
29706991612,3cyclictest5340-21apt20:35:002
29706991612,3cyclictest29694-21cyclictest21:54:472
2970499166,2cyclictest141rcu_preempt23:15:111
2970499160,16cyclictest0-21swapper/121:17:281
2970499160,15cyclictest0-21swapper/123:14:031
2970499160,0cyclictest0-21swapper/122:20:261
2970099162,12cyclictest3389-21grep19:20:160
2970099161,14cyclictest537-21snmpd21:00:130
2970099160,15cyclictest0-21swapper/021:05:140
2970099160,15cyclictest0-21swapper/021:05:140
2970099160,13cyclictest0-21swapper/020:55:130
2970999151,14cyclictest0-21swapper/323:15:503
29706991514,1cyclictest0-21swapper/200:39:592
29706991513,1cyclictest0-21swapper/221:49:412
29706991510,3cyclictest6302-21tail23:35:162
2970699150,15cyclictest0-21swapper/221:11:402
2970699150,14cyclictest0-21swapper/222:39:042
2970499158,6cyclictest1201-21awk00:10:221
2970499157,7cyclictest7424-21grep23:35:251
2970499152,2cyclictest141rcu_preempt22:09:051
29704991514,1cyclictest0-21swapper/100:38:391
29704991513,1cyclictest25-21ksoftirqd/121:55:481
29704991510,4cyclictest1889-21wc20:25:191
2970499150,1cyclictest0-21swapper/121:36:241
2970499150,15cyclictest22437-21diskmemload00:24:031
2970499150,15cyclictest22437-21diskmemload00:24:031
2970099153,10cyclictest11064-21grep19:35:260
2970099150,14cyclictest0-21swapper/020:45:170
2970999146,3cyclictest29936-21apt-get19:10:123
2970999146,3cyclictest29936-21apt-get19:10:123
29709991411,2cyclictest4388-21head22:55:163
29709991410,3cyclictest2936-21sshd22:15:203
29706991410,3cyclictest24936-21grep00:25:152
29706991410,3cyclictest17161-21grep00:20:192
29706991410,3cyclictest17161-21grep00:20:192
29706991410,3cyclictest15781-21grep19:45:252
29706991410,2cyclictest0-21swapper/223:30:062
2970699140,14cyclictest0-21swapper/222:55:392
2970499149,3cyclictest25-21ksoftirqd/123:01:071
2970499148,5cyclictest29570-21kworker/u8:119:28:001
2970499142,1cyclictest141rcu_preempt21:23:171
29704991411,2cyclictest32037-21sh19:15:011
29704991411,2cyclictest32037-21sh19:15:001
29704991411,2cyclictest14022-21wc19:45:001
29704991410,3cyclictest25-21ksoftirqd/121:35:011
29704991410,3cyclictest12480-21sh20:50:001
2970499140,1cyclictest0-21swapper/120:34:201
2970499140,14cyclictest0-21swapper/123:30:441
2970499140,0cyclictest141rcu_preempt23:55:061
2970099148,5cyclictest28600-21wget20:15:130
2970099148,5cyclictest22375-21cat20:00:190
2970099148,5cyclictest17271-21tail19:50:180
2970099141,12cyclictest0-21swapper/020:50:210
2970099140,13cyclictest0-21swapper/020:10:130
2970099140,12cyclictest0-21swapper/020:30:250
2970999139,2cyclictest25236-21grep21:50:133
2970999131,4cyclictest0-21swapper/322:40:213
29709991313,0cyclictest0-21swapper/323:55:123
29709991310,2cyclictest24489-21ssh21:49:593
29709991310,2cyclictest19945-21grep22:45:173
29709991310,2cyclictest16830-21turbostat00:40:013
29709991310,2cyclictest14033-21cat23:20:173
2970999130,1cyclictest0-21swapper/321:31:183
2970999130,12cyclictest26603-21sh22:10:153
2970699139,3cyclictest31356-21chrt19:10:252
2970699139,3cyclictest31356-21chrt19:10:242
2970699139,3cyclictest10102-21cron20:45:002
2970699138,4cyclictest12052-21ssh23:19:472
2970699136,2cyclictest225-21systemd-journal21:00:002
29706991310,2cyclictest7794-21head19:30:162
29706991310,2cyclictest6239-21grep23:55:002
29706991310,2cyclictest3957-21grep19:20:242
29706991310,2cyclictest23278-21wc23:45:182
29706991310,2cyclictest17513-21ls19:50:202
29706991310,2cyclictest12885-21grep20:50:142
2970699130,6cyclictest0-21swapper/223:09:182
2970499139,3cyclictest9294-21sed00:15:191
2970499139,3cyclictest27766-21head20:10:271
2970499138,4cyclictest21036-21tr23:05:171
2970499137,5cyclictest4266-21cut22:35:221
2970499137,4cyclictest17435-21cron21:00:001
2970499137,2cyclictest16808-21ssh21:25:201
29704991312,1cyclictest0-21swapper/121:47:001
2970499131,10cyclictest15437-21sensors23:40:231
29704991310,2cyclictest2938-21cat22:15:191
29704991310,2cyclictest20989-21sort21:05:141
29704991310,2cyclictest20989-21sort21:05:131
2970499130,1cyclictest141rcu_preempt22:45:471
2970999129,2cyclictest9499-21tr21:40:203
2970999129,2cyclictest9252-21cat19:35:013
2970999129,2cyclictest30790-21sh23:30:183
2970999129,2cyclictest28892-21tr23:10:133
2970999129,2cyclictest2394-21wc22:35:003
2970999129,2cyclictest20200-21wc21:05:013
2970999129,2cyclictest15669-21grep00:00:153
2970999129,2cyclictest13981-21cat19:45:003
2970999129,2cyclictest10932-21grep22:20:163
2970999128,3cyclictest7080-21ssh21:39:303
2970999128,3cyclictest1710-21head21:55:153
2970999120,2cyclictest0-21swapper/322:38:013
2970999120,11cyclictest31628-21expr23:50:173
2970699129,2cyclictest27857-21cat22:30:192
2970699129,2cyclictest2311-21head21:55:202
2970699129,2cyclictest1995-21cron19:20:002
2970699129,2cyclictest18691-21tr22:25:142
2970699129,2cyclictest17248-21grep21:25:252
2970699129,2cyclictest1126-21grep20:25:142
2970699128,1cyclictest19500-21df_inode19:55:172
29706991211,1cyclictest0-21swapper/221:18:012
2970699120,12cyclictest0-21swapper/221:44:182
2970499129,2cyclictest7214-21grep20:35:251
2970499129,2cyclictest29224-21tail20:15:161
2970499128,3cyclictest25068-21wget21:50:131
2970499127,2cyclictest25-21ksoftirqd/119:59:591
2970499124,5cyclictest25-21ksoftirqd/119:15:201
29704991210,1cyclictest9660-21grep19:35:151
29704991210,1cyclictest23726-21grep20:05:121
2970499120,2cyclictest0-21swapper/122:34:481
2970099129,2cyclictest1084-21grep20:25:140
2970099120,11cyclictest0-21swapper/019:10:150
2970099120,11cyclictest0-21swapper/019:10:140
2970999118,2cyclictest733-21grep21:15:253
2970999118,2cyclictest6354-21grep19:25:253
2970999118,2cyclictest420-21chrt20:23:263
2970999118,2cyclictest4022-21rm00:31:313
2970999118,2cyclictest3969-21tail20:30:153
2970999118,2cyclictest318-21grep00:10:163
2970999118,2cyclictest25267-21ssh00:05:263
2970999118,2cyclictest25050-21ssh21:10:313
2970999118,2cyclictest24950-21tr00:25:143
2970999118,2cyclictest21696-21cat20:00:123
2970999118,2cyclictest17771-21ssh21:25:373
2970999118,2cyclictest17108-21rm23:41:263
2970999118,2cyclictest13865-21ssh23:00:243
2970999118,2cyclictest11539-21systemctl19:40:003
2970999118,2cyclictest10655-21ssh22:00:213
2970999116,4cyclictest14428-21tr19:45:143
2970999114,3cyclictest225-21systemd-journal21:00:003
2970999110,10cyclictest30353-21taskset20:16:433
2970699118,2cyclictest6761-21cron19:30:002
2970699118,2cyclictest32178-21ssh21:35:052
2970699118,2cyclictest30844-21taskset20:20:002
2970699118,2cyclictest26458-21ssh21:30:582
2970699118,2cyclictest19752-21ssh23:24:172
2970699118,2cyclictest18165-21diskstats22:05:152
2970699118,2cyclictest17470-21ssh23:03:142
2970699118,2cyclictest10834-21chrt22:40:052
2970699110,10cyclictest28503-21head22:50:162
2970699110,10cyclictest26041-21cat20:10:122
2970699110,10cyclictest0-21swapper/223:26:052
2970499118,2cyclictest9198-21ssh22:00:101
2970499118,2cyclictest8895-21chrt19:31:551
2970499118,2cyclictest6491-21ssh22:56:091
2970499118,2cyclictest391-21chrt20:23:031
2970499118,2cyclictest241rcuc/120:50:261
2970499117,3cyclictest3033-21grep19:20:141
2970499117,3cyclictest25994-21ssh23:27:371
2970499111,2cyclictest141rcu_preempt21:40:121
193812110,1sleep20-21swapper/223:43:142
2970999108,1cyclictest393-21sed19:15:143
2970999108,1cyclictest2313-21basename19:20:123
2970999108,1cyclictest20793-21chrt19:57:063
2970999108,1cyclictest20220-21sh23:05:123
2970999108,1cyclictest18498-21seq19:53:053
2970999108,1cyclictest17428-21rm22:05:083
2970999108,1cyclictest11682-21gzip20:45:213
2970999107,2cyclictest5795-21grep20:35:133
2970999107,2cyclictest18344-21ssh22:25:093
2970999107,2cyclictest14868-21awk20:55:003
2970999107,2cyclictest0-21swapper/321:05:123
2970999107,2cyclictest0-21swapper/321:05:113
2970999106,3cyclictest24801-21cut20:05:193
2970699108,2cyclictest23801-21awk20:05:132
2970699108,1cyclictest308662sleep220:20:042
2970699108,1cyclictest221342sleep221:06:252
2970699108,1cyclictest221342sleep221:06:242
2970699107,2cyclictest21356-21aten2_r7power_e20:00:112
2970699107,2cyclictest10998-21grep20:45:152
2970699107,2cyclictest0-21swapper/221:00:172
2970699103,3cyclictest5606-21apache_volume20:35:122
2970699100,9cyclictest135862sleep219:40:572
2970499108,2cyclictest0-21swapper/119:48:211
2970499108,1cyclictest26003-21ssh21:11:161
2970499107,2cyclictest21533-21sort20:00:131
2970499107,2cyclictest16835-21tr19:50:151
297099997,1cyclictest8000-21apache_processe20:40:123
297099997,1cyclictest26390-21expr20:10:153
297099997,1cyclictest1151-21expr20:25:143
297069996,2cyclictest9649-21tr19:35:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional