You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 08:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Apr 22, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
134029171,16sleep30-21swapper/319:07:563
117728874,9sleep00-21swapper/019:05:530
139828744,39sleep20-21swapper/219:08:392
141727857,6sleep10-21swapper/119:08:551
59732550,0sleep00-21swapper/000:10:190
94522510,0sleep10-21swapper/100:31:401
258572480,0sleep00-21swapper/023:43:580
1582992521,3cyclictest23192-21grep21:00:171
158699240,23cyclictest0-21swapper/221:52:082
159199232,0cyclictest141rcu_preempt23:11:003
158699230,23cyclictest0-21swapper/219:48:062
158699230,23cyclictest0-21swapper/200:16:162
158699230,0cyclictest0-21swapper/223:32:012
159199220,1cyclictest0-21swapper/300:20:143
158699220,1cyclictest0-21swapper/200:24:402
158699220,0cyclictest0-21swapper/222:05:252
1582992221,1cyclictest0-21swapper/122:59:411
159199210,2cyclictest0-21swapper/321:35:153
158699210,21cyclictest0-21swapper/222:32:482
158699210,1cyclictest0-21swapper/223:35:392
158699210,1cyclictest0-21swapper/223:16:402
158699210,1cyclictest0-21swapper/221:35:042
159199203,0cyclictest141rcu_preempt23:02:023
159199202,0cyclictest141rcu_preempt21:40:243
1591992018,1cyclictest39-21ksoftirqd/321:15:543
158699200,1cyclictest0-21swapper/222:18:342
158699200,1cyclictest0-21swapper/221:20:412
1582992019,1cyclictest0-21swapper/100:15:211
1591991917,1cyclictest1142-21runrttasks22:25:103
1591991917,1cyclictest0-21swapper/323:30:453
158699190,1cyclictest0-21swapper/200:06:322
158699190,19cyclictest0-21swapper/221:31:112
158699190,0cyclictest0-21swapper/222:46:372
158299191,18cyclictest27015-21diskmemload21:34:201
159199182,0cyclictest141rcu_preempt23:21:563
1591991816,1cyclictest0-21swapper/323:35:143
1591991816,1cyclictest0-21swapper/322:08:473
1591991816,1cyclictest0-21swapper/321:29:573
158699180,1cyclictest0-21swapper/223:03:532
158699180,1cyclictest0-21swapper/200:02:462
158699180,0cyclictest0-21swapper/223:06:352
1582991817,1cyclictest0-21swapper/123:31:531
1582991817,1cyclictest0-21swapper/122:37:381
158299181,3cyclictest0-21swapper/123:50:121
158299180,7cyclictest27015-21diskmemload22:20:451
157799180,18cyclictest0-21swapper/021:50:490
159199171,5cyclictest0-21swapper/323:55:133
1591991713,1cyclictest39-21ksoftirqd/322:46:173
158699170,17cyclictest0-21swapper/223:42:182
158699170,0cyclictest0-21swapper/200:10:152
158299177,8cyclictest28699-21/usr/sbin/munin20:05:131
158299173,14cyclictest0-21swapper/123:18:221
1582991716,1cyclictest0-21swapper/123:36:541
158299170,14cyclictest0-21swapper/100:01:231
157799176,9cyclictest3143-21sh20:20:010
157799174,9cyclictest1235-21ntpd23:45:080
157799174,7cyclictest16876-21grep23:00:000
157799171,9cyclictest18771-21munin-run23:40:000
90852160,0chrt0-21swapper/000:31:220
159199164,1cyclictest0-21swapper/322:57:393
159199161,5cyclictest39-21ksoftirqd/322:30:143
1591991614,1cyclictest39-21ksoftirqd/321:15:003
1591991614,1cyclictest39-21ksoftirqd/320:05:223
1591991610,5cyclictest39-21ksoftirqd/319:55:213
159199160,3cyclictest141rcu_preempt19:15:163
159199160,1cyclictest0-21swapper/300:30:133
158699162,2cyclictest0-21swapper/221:17:542
158699160,0cyclictest0-21swapper/223:12:242
1582991614,1cyclictest0-21swapper/121:46:571
158299161,1cyclictest13634-21taskset19:32:011
157799163,6cyclictest19813-21munin-run20:55:010
157799161,15cyclictest0-21swapper/022:28:030
157799160,15cyclictest0-21swapper/021:15:150
64612150,1chrt0-21swapper/023:51:070
159199156,6cyclictest12792-21cron00:15:013
159199152,1cyclictest141rcu_preempt23:44:373
1586991512,1cyclictest0-21swapper/223:20:162
158699150,2cyclictest0-21swapper/221:10:272
158699150,15cyclictest0-21swapper/222:59:412
158699150,15cyclictest0-21swapper/221:48:202
158699150,11cyclictest0-21swapper/200:30:032
1582991513,1cyclictest3952-21cpuspeed_turbos21:15:161
158299151,1cyclictest9536-21sendmail_mailtr20:30:221
158299151,0cyclictest141rcu_preempt20:45:141
158299150,2cyclictest0-21swapper/120:10:151
157799157,2cyclictest225-21systemd-journal22:04:150
157799151,8cyclictest4683-21apt-get21:35:150
157799151,13cyclictest3214-21ssh23:30:150
159199149,2cyclictest11413-21cron23:54:593
159199144,4cyclictest141rcu_preempt22:00:143
159199142,6cyclictest39-21ksoftirqd/323:27:253
1591991411,2cyclictest39-21ksoftirqd/300:15:213
159199140,13cyclictest0-21swapper/321:54:433
158699145,8cyclictest0-21swapper/221:43:192
158699141,1cyclictest0-21swapper/223:52:112
1586991411,2cyclictest24125-21grep21:00:262
1586991410,3cyclictest17417-21sed22:20:272
158699140,14cyclictest0-21swapper/223:48:042
158299141,13cyclictest27015-21diskmemload00:26:581
1582991411,2cyclictest9689-21ssh23:15:001
158299140,2cyclictest0-21swapper/121:50:081
158299140,2cyclictest0-21swapper/121:13:121
158299140,14cyclictest0-21swapper/122:41:151
1577991414,0cyclictest0-21swapper/023:04:310
1577991410,3cyclictest19827-21idleruntime-cro21:24:590
1577991410,3cyclictest18391-21grep23:20:140
2204121310,2sleep022273-21sed21:25:260
159199138,4cyclictest39-21ksoftirqd/300:08:173
1591991311,1cyclictest0-21swapper/322:37:363
1591991310,2cyclictest28364-21cut23:45:173
1591991310,2cyclictest16843-21tr22:20:233
1591991310,2cyclictest10793-21grep19:25:263
159199130,8cyclictest0-21swapper/321:55:143
158699139,3cyclictest32135-21ls22:10:192
158699139,3cyclictest22096-21ls22:05:002
158699138,3cyclictest20340-21ssh22:41:562
158699132,8cyclictest23844-21apt-get22:25:152
1586991310,2cyclictest8508-21sh22:54:592
1586991310,2cyclictest8234-21awk22:35:152
158699130,13cyclictest0-21swapper/223:29:012
158299139,3cyclictest7570-21sh22:35:001
158299139,3cyclictest27027-21basename21:09:591
158299138,2cyclictest0-21swapper/123:03:271
158299131,11cyclictest0-21swapper/123:55:111
1582991311,1cyclictest0-21swapper/122:28:401
158299130,6cyclictest38150irq/131-snd_hda19:45:131
158299130,4cyclictest0-21swapper/122:09:491
158299130,1cyclictest27015-21diskmemload23:47:141
158299130,1cyclictest0-21swapper/122:45:541
158299130,13cyclictest0-21swapper/122:17:001
158299130,12cyclictest19444-21taskset20:50:351
157799135,7cyclictest27016-21munin-run21:09:590
157799132,3cyclictest0-21swapper/000:08:160
1577991311,2cyclictest13-21ksoftirqd/021:57:420
1577991310,2cyclictest29250-21grep21:49:590
1577991310,2cyclictest24320-21sh22:45:000
1577991310,2cyclictest22884-21grep00:20:200
1577991310,2cyclictest12739-21sed23:55:150
159199129,2cyclictest8963-21cut20:30:173
159199129,2cyclictest7047-21tr19:20:143
159199129,2cyclictest5533-21cron20:25:013
159199129,2cyclictest29021-21sh23:06:243
159199129,2cyclictest28424-21cron20:05:003
159199129,2cyclictest21787-21tail20:55:273
159199129,2cyclictest15549-21tr20:45:143
159199129,2cyclictest13701-21tail20:40:163
159199126,5cyclictest39-21ksoftirqd/300:04:383
159199126,4cyclictest27015-21diskmemload22:43:583
159199124,6cyclictest27015-21diskmemload21:48:233
159199121,1cyclictest141rcu_preempt22:16:353
158699129,2cyclictest8907-21cron19:25:002
158699129,2cyclictest7903-21cron20:30:012
158699129,2cyclictest4958-21tail19:15:172
158699129,2cyclictest28444-21perf20:05:002
158699129,2cyclictest23219-21sed19:50:262
158699129,2cyclictest21762-21grep21:25:222
158699129,2cyclictest18849-21cron19:44:592
158699129,2cyclictest14136-21ls20:40:202
158699129,2cyclictest12690-21grep23:55:152
158699128,3cyclictest22227-21sh21:00:002
1586991210,1cyclictest32-21ksoftirqd/219:30:002
158299129,2cyclictest7595-21head19:20:171
158299129,2cyclictest29487-21grep00:05:181
158299129,2cyclictest27745-21grep23:05:271
158299129,2cyclictest21630-21grep21:25:201
158299129,2cyclictest18228-21cat19:40:261
158299129,2cyclictest14368-21tr00:35:161
158299129,2cyclictest12365-21wc21:20:151
158299121,2cyclictest22229-21latency_hist20:59:591
157799129,2cyclictest32158-21grep20:10:200
157799129,2cyclictest32136-21sort22:10:190
157799129,2cyclictest29468-21sort21:30:150
157799129,2cyclictest24579-21sort19:55:160
157799129,2cyclictest22798-21tr21:00:150
157799129,2cyclictest2257-21grep23:10:150
157799129,2cyclictest1687-21grep22:30:250
157799129,2cyclictest14818-21sort19:35:170
157799129,2cyclictest1267-21wc23:09:590
157799129,2cyclictest10276-21cat19:25:190
157799128,3cyclictest14788-21gpgv22:20:010
157799128,3cyclictest11473-21ssh23:15:230
157799124,7cyclictest30085-21grep00:25:140
159199119,1cyclictest18997-21chrt00:37:383
159199118,2cyclictest4156-21ssh22:51:343
159199118,2cyclictest31570-21ssh22:10:163
159199118,2cyclictest29693-21ssh00:25:003
159199118,2cyclictest23231-21grep21:00:173
159199118,2cyclictest2249-21ssh21:33:323
159199118,2cyclictest22325-21sed19:50:163
159199118,2cyclictest18244-21sed20:50:153
159199118,2cyclictest1799-21cut20:15:173
159199118,2cyclictest17468-21grep19:40:163
159199118,2cyclictest12255-21head20:35:283
159199118,2cyclictest10518-21ssh23:15:153
159199113,4cyclictest225-21systemd-journal21:21:053
158699118,2cyclictest9831-21sort19:25:162
158699118,2cyclictest7976-21ssh21:55:322
158699118,2cyclictest31671-21sort20:10:172
158699118,2cyclictest19419-21users20:50:272
158699118,2cyclictest16827-21ssh00:35:552
158699118,2cyclictest16247-21sh19:40:002
158699118,2cyclictest11321-21grep20:35:182
158699110,11cyclictest0-21swapper/220:09:162
158299118,3cyclictest10110-21sh21:57:121
158299118,2cyclictest4465-21ssh21:35:051
158299118,2cyclictest16960-21ssh22:01:011
158299117,1cyclictest41582sleep119:15:041
158299116,1cyclictest19336-21ssh23:40:141
158299114,6cyclictest32162-21/usr/sbin/munin22:10:201
158299113,4cyclictest3576-21apache200:22:401
158299111,10cyclictest10372-21ssh00:13:051
158299110,7cyclictest141rcu_preempt20:40:221
157799118,2cyclictest9814-21ssh22:35:260
157799118,2cyclictest8918-21head20:30:160
157799118,2cyclictest642-21ssh22:50:110
157799118,2cyclictest6178-21expr20:25:150
157799118,2cyclictest32646-21ssh21:12:430
157799118,2cyclictest32316-21ssh23:28:180
157799118,2cyclictest29959-21ssh22:47:560
157799118,2cyclictest24996-21chrt22:05:450
157799118,2cyclictest22886-21cat19:50:220
157799118,2cyclictest21149-21ssh00:39:200
157799118,2cyclictest18472-21ssh21:43:120
157799118,2cyclictest16032-21head20:45:150
157799118,2cyclictest16013-21chrt00:16:000
157799118,2cyclictest15344-21ssh22:20:110
157799118,2cyclictest14062-21wc19:34:590
157799117,2cyclictest4268-21grep19:15:150
157799115,1cyclictest30827-21perf20:09:590
157799113,5cyclictest20239-21apache_accesses00:00:120
157799113,4cyclictest225-21systemd-journal20:40:010
257402101,1sleep2225-21systemd-journal19:56:212
159199107,2cyclictest5860-21aten2_r7power_e20:25:153
159199107,2cyclictest25724-21basename21:05:173
159199107,2cyclictest19269-21wc19:45:143
159199107,2cyclictest1883-21cut19:10:133
159199107,2cyclictest14145-21apt-get19:35:143
159199102,5cyclictest0-21swapper/319:30:023
158699108,1cyclictest9756-21cat20:30:252
158699108,1cyclictest3574-21grep20:20:152
158699108,1cyclictest16452-21ntp_kernel_pll_20:45:202
158699108,1cyclictest1095-21basename20:15:142
158699107,2cyclictest29965-21ssh00:25:122
158699107,2cyclictest2278-21ls19:10:142
158699100,9cyclictest25017-21grep21:05:142
158299108,2cyclictest28513-21taskset23:25:261
158299108,2cyclictest27793-21cat20:00:261
158299108,2cyclictest22708-21ntpq19:50:211
158299108,1cyclictest9477-21wc19:25:141
158299108,1cyclictest4802-21tr20:20:231
158299108,1cyclictest25848-21taskset19:57:431
158299108,1cyclictest10619-21wget20:35:151
158299107,2cyclictest19157-21grep23:20:181
158299107,2cyclictest12954-21apt-get21:40:131
158299100,9cyclictest5194-21ssh22:52:231
157799108,1cyclictest3702-21taskset19:12:140
157799108,1cyclictest3486-21grep20:20:130
157799107,2cyclictest7827-21cut19:20:180
157799107,2cyclictest26381-21cut20:00:150
157799107,2cyclictest20760-21sed19:45:240
15919996,2cyclictest31455-21expr20:10:153
15829997,1cyclictest29182sleep120:17:141
15829997,1cyclictest1851-21wget19:10:131
15829997,1cyclictest14457-21seq19:35:151
15829996,2cyclictest6808-21cat20:25:211
15779997,1cyclictest16848-21cut19:40:140
15779996,2cyclictest13086-21grep20:40:140
15779986,1cyclictest20417-21ls20:55:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional