You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-07 - 04:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu May 07, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1975210164,33sleep20-21swapper/219:07:342
208029568,9sleep30-21swapper/319:08:563
195329348,40sleep00-21swapper/019:07:160
211828564,6sleep10-21swapper/119:09:261
224699260,18cyclictest0-21swapper/023:00:400
225399234,8cyclictest0-21swapper/220:15:132
225399232,1cyclictest0-21swapper/223:07:522
224699231,1cyclictest0-21swapper/000:03:590
2256992221,1cyclictest39-21ksoftirqd/322:43:343
225099220,2cyclictest0-21swapper/100:29:081
225099220,22cyclictest0-21swapper/121:43:541
225099220,19cyclictest0-21swapper/121:30:461
225399210,21cyclictest0-21swapper/222:38:532
224699210,2cyclictest0-21swapper/022:07:550
225099205,10cyclictest0-21swapper/100:35:111
225099200,19cyclictest0-21swapper/121:47:151
2246992019,1cyclictest0-21swapper/023:12:590
224699200,2cyclictest0-21swapper/000:31:200
225399190,19cyclictest0-21swapper/221:20:582
225099192,14cyclictest27851-21diskmemload21:19:331
2253991816,1cyclictest0-21swapper/222:56:152
2250991817,1cyclictest0-21swapper/122:13:011
2253991715,1cyclictest32-21ksoftirqd/223:48:272
225399170,17cyclictest0-21swapper/222:52:172
2250991717,0cyclictest0-21swapper/122:31:431
2250991716,1cyclictest0-21swapper/122:42:191
225099171,10cyclictest6429-21cron23:50:001
225099170,17cyclictest0-21swapper/123:08:161
225099170,16cyclictest0-21swapper/123:01:341
225099170,16cyclictest0-21swapper/121:53:331
224699170,17cyclictest0-21swapper/023:18:440
2253991613,1cyclictest27851-21diskmemload00:34:122
225399160,16cyclictest0-21swapper/200:37:122
2250991616,0cyclictest0-21swapper/122:18:151
2250991611,1cyclictest141rcu_preempt23:38:001
225099160,1cyclictest25502-21/usr/sbin/munin00:20:201
225099160,1cyclictest0-21swapper/100:14:381
225699152,4cyclictest0-21swapper/321:15:133
225699152,12cyclictest27851-21diskmemload21:46:593
225399154,10cyclictest8584-21awk20:30:002
2253991512,2cyclictest32-21ksoftirqd/223:05:002
2253991511,2cyclictest32-21ksoftirqd/223:40:122
225399150,7cyclictest0-21swapper/200:29:372
225399150,13cyclictest0-21swapper/200:19:132
2250991511,2cyclictest0-21swapper/121:59:411
224699155,5cyclictest5113-21idleruntime-cro23:30:000
224699152,11cyclictest0-21swapper/021:15:140
224699151,7cyclictest3935-21munin-run21:14:590
224699150,1cyclictest0-21swapper/023:51:530
224699150,1cyclictest0-21swapper/022:04:100
225699148,2cyclictest11817-21sed22:35:253
225699145,5cyclictest15685-21cut23:55:143
2256991414,0cyclictest0-21swapper/323:09:543
225699141,10cyclictest16924-21ssh22:20:093
225699141,10cyclictest16924-21ssh22:20:093
2256991410,3cyclictest27047-21wc21:05:223
225699140,8cyclictest0-21swapper/321:40:133
225399144,7cyclictest16717-21munin-run19:40:012
225399142,4cyclictest0-21swapper/200:00:142
225399141,12cyclictest27851-21diskmemload21:36:572
2253991411,2cyclictest21217-21grep19:45:262
225399140,12cyclictest0-21swapper/221:42:512
225399140,12cyclictest0-21swapper/221:18:192
2250991410,3cyclictest10221-21tr22:35:131
2250991410,2cyclictest22556-21grep20:55:251
225099140,0cyclictest0-21swapper/100:31:391
224699143,9cyclictest1487-21apt22:30:130
224699143,6cyclictest3856-21basename20:20:000
224699142,3cyclictest141rcu_preempt23:06:270
224699141,7cyclictest15738-21munin-run20:45:000
224699140,14cyclictest0-21swapper/000:18:050
224699140,13cyclictest23028-21cpuspeed_turbos21:45:140
225699139,2cyclictest5439-21grep23:10:193
225699138,3cyclictest0-21swapper/319:40:133
225699134,6cyclictest0-21swapper/323:00:133
2256991310,2cyclictest4598-21sleep19:14:593
2256991310,2cyclictest29601-21grep21:10:203
2256991310,2cyclictest23727-21tr23:40:203
2256991310,2cyclictest1822-21sh00:25:133
2256991310,2cyclictest11912-21grep22:55:173
225699130,1cyclictest0-21swapper/300:08:053
225699130,12cyclictest6199-21cron20:25:003
225699130,12cyclictest25474-21tr22:25:133
225699130,12cyclictest0-21swapper/300:31:473
225399133,1cyclictest141rcu_preempt21:46:022
2253991310,2cyclictest9117-21tr20:30:142
2253991310,2cyclictest8494-21tr21:55:202
2253991310,2cyclictest31715-21grep20:10:142
2253991310,2cyclictest30324-21cut23:25:182
2253991310,2cyclictest17767-21head19:40:162
2253991310,2cyclictest12319-21grep20:35:212
225399130,2cyclictest12297-21kworker/u8:223:20:522
225399130,12cyclictest8062-21cat22:15:002
225399130,12cyclictest24480-21tr21:00:222
225399130,12cyclictest0-21swapper/222:07:022
225099139,3cyclictest18307-21sh20:50:001
225099139,3cyclictest11500-21tr22:55:161
225099138,4cyclictest27878-21chrt21:28:491
2250991313,0cyclictest0-21swapper/121:38:311
2250991310,2cyclictest31424-21sleep20:10:001
2250991310,2cyclictest24906-21sed22:05:151
2250991310,2cyclictest16696-21cron19:40:011
2250991310,2cyclictest14369-21sed20:40:151
225099130,12cyclictest0-21swapper/123:51:401
224699139,3cyclictest26948-21grep21:05:220
224699139,3cyclictest19846-21tr19:45:140
224699139,2cyclictest31717-21tr20:10:130
224699138,3cyclictest15304-21wc22:00:010
224699134,3cyclictest452-21systemd-logind22:15:220
224699131,9cyclictest6426-21cron23:50:000
224699131,6cyclictest10707-21munin-run22:55:000
2246991311,1cyclictest0-21swapper/022:49:490
2246991310,2cyclictest7350-21grep21:35:210
2246991310,2cyclictest4868-21sed20:20:160
2246991310,2cyclictest31369-21cut21:50:130
2246991310,2cyclictest1341-21tail22:10:180
224699130,12cyclictest0-21swapper/000:27:570
225699129,2cyclictest9978-21ntpq20:30:203
225699129,2cyclictest8251-21head19:20:173
225699129,2cyclictest8075-21grep21:55:173
225699129,2cyclictest6696-21grep19:15:253
225699129,2cyclictest29929-21tr23:25:163
225699129,2cyclictest27942-21cat22:45:193
225699129,2cyclictest25893-21cat19:55:253
225699129,2cyclictest24799-21cut22:05:163
225699129,2cyclictest20534-21wc19:45:193
225699129,2cyclictest17829-21sed20:45:263
225699129,2cyclictest17144-21tr00:15:143
225699129,2cyclictest15953-21sed23:35:253
225699129,2cyclictest14962-21grep21:20:253
225699128,1cyclictest20024-21python00:35:233
225699127,2cyclictest273-21systemd-udevd23:52:043
225699121,10cyclictest1436-21sendmail-mta23:23:553
225699120,4cyclictest13220-21kworker/u8:221:39:393
225699120,1cyclictest0-21swapper/321:28:233
225699120,11cyclictest14329-21head20:40:153
225699120,11cyclictest12996-21ssh23:34:363
225399129,2cyclictest9348-21grep00:10:182
225399129,2cyclictest7634-21sort23:50:162
225399129,2cyclictest353-21grep00:05:152
225399129,2cyclictest32345-21wc21:50:192
225399129,2cyclictest32-21ksoftirqd/221:10:212
225399129,2cyclictest31601-21cat20:10:002
225399129,2cyclictest27215-21chrt21:28:182
225399129,2cyclictest26007-21grep19:55:262
225399129,2cyclictest23080-21readlink21:00:002
225399129,2cyclictest15735-21ls20:45:002
225399129,2cyclictest12023-21wc19:30:002
225399128,3cyclictest32-21ksoftirqd/221:34:512
2253991210,1cyclictest19686-21sed22:40:212
225399120,11cyclictest0-21swapper/222:25:152
225099129,2cyclictest9572-21cron19:24:591
225099129,2cyclictest4683-21tail22:50:251
225099129,2cyclictest32286-21grep00:05:121
225099129,2cyclictest27333-21sed21:05:261
225099129,2cyclictest19845-21grep19:45:141
225099129,2cyclictest17237-21grep23:55:261
225099129,2cyclictest1420-21cron20:15:001
225099129,2cyclictest11508-21tr20:35:141
225099128,3cyclictest17776-21tail22:20:161
225099128,3cyclictest17776-21tail22:20:151
225099128,2cyclictest24362-21grep23:40:251
225099127,4cyclictest27592-21ssh00:01:511
225099127,1cyclictest2237-21cyclictest00:18:351
224699129,2cyclictest9689-21wc19:25:000
224699129,2cyclictest8893-21basename20:30:130
224699129,2cyclictest22699-21grep19:50:140
224699129,2cyclictest20267-21head20:50:270
224699129,2cyclictest15522-21wc23:35:220
224699128,3cyclictest31410-21cron20:10:000
224699122,1cyclictest0-21swapper/021:22:000
2246991210,1cyclictest8292-21ssh00:10:050
2246991210,1cyclictest2237-21cyclictest23:32:130
224699120,10cyclictest0-21swapper/022:27:000
225699119,1cyclictest22721-21sed19:50:143
225699118,2cyclictest7712-21tr20:25:213
225699118,2cyclictest4941-21ssh23:48:523
225699118,2cyclictest31964-21ssh00:23:523
225699118,2cyclictest31605-21ssh21:50:143
225699118,2cyclictest30182-21if_enp2s020:05:183
225699118,2cyclictest2662-21cat20:15:203
225699118,2cyclictest2622-21ssh22:50:053
225699118,2cyclictest2548-21ssh22:10:443
225699118,2cyclictest23554-21sh00:00:023
225699118,2cyclictest13344-21ssh22:17:363
225699118,2cyclictest1318-21taskset20:13:463
225699118,2cyclictest13107-21chrt20:37:193
225699118,2cyclictest13045-21expr23:15:153
225699118,2cyclictest12863-21grep19:30:173
225699118,2cyclictest11511-21ssh00:11:143
225699118,2cyclictest11099-21wc19:25:223
225699117,3cyclictest5605-21date21:35:003
225699112,5cyclictest27851-21diskmemload22:00:153
225399118,2cyclictest6688-21grep19:15:252
225399118,2cyclictest5718-21sed20:20:252
225399118,2cyclictest5675-21rm23:10:222
225399118,2cyclictest5074-21ssh22:31:392
225399118,2cyclictest2530-21cut19:10:142
225399118,2cyclictest21766-21ssh23:58:502
225399118,2cyclictest19358-21grep20:50:172
225399118,2cyclictest18521-21mailstats22:20:212
225399118,2cyclictest18521-21mailstats22:20:202
225399118,2cyclictest17532-21chrt23:17:462
225399118,2cyclictest16233-21ssh22:00:162
225399118,2cyclictest14556-21ssh23:35:142
225399118,2cyclictest11711-21ssh22:16:202
225399118,2cyclictest1038-21wc00:25:002
225399117,3cyclictest17420-21sed20:45:212
225399110,10cyclictest30623-21taskset22:46:542
225099119,1cyclictest5787-21grep19:15:151
225099119,1cyclictest13899-21cut21:20:161
225099118,2cyclictest6158-21taskset20:24:311
225099118,2cyclictest4486-21expr23:10:121
225099118,2cyclictest27678-21sed20:00:171
225099118,2cyclictest24683-21systemctl21:00:231
225099118,2cyclictest24429-21grep19:55:131
225099118,2cyclictest23708-21ssh23:20:511
225099118,2cyclictest23552-21sed19:50:231
225099118,2cyclictest19925-21ssh22:02:101
225099118,2cyclictest10523-21sort19:25:171
225099118,2cyclictest1014-21wc22:30:001
225099117,3cyclictest8724-21sh23:31:171
225099117,3cyclictest29212-21grep21:10:171
225099117,2cyclictest6711-21tr20:25:151
225099110,10cyclictest31017-21sh22:47:131
225099110,10cyclictest2666-21sh23:28:061
225099110,10cyclictest15451-21ssh23:16:091
225099110,10cyclictest0-21swapper/120:15:131
224699118,2cyclictest9983-21grep19:25:160
224699118,2cyclictest8588-21munin-run20:30:010
224699118,2cyclictest32653-21ssh00:05:130
224699118,2cyclictest31041-21grep21:30:180
224699118,2cyclictest27688-21head20:00:170
224699118,2cyclictest26913-21ls00:20:210
224699118,2cyclictest26022-21sed19:55:260
224699118,2cyclictest25708-21ssh22:44:370
224699118,2cyclictest25089-21ssh23:21:550
224699118,2cyclictest23215-21grep23:40:160
224699118,2cyclictest22347-21sed21:25:170
224699118,2cyclictest20532-21ssh00:35:310
224699118,2cyclictest18863-21taskset19:42:070
224699118,2cyclictest14392-21ssh21:40:130
224699118,2cyclictest12832-21grep20:35:260
224699118,2cyclictest11887-21cut22:55:160
224699118,2cyclictest10055-21ssh22:35:130
224699118,1cyclictest16958-21apt-get22:20:110
224699118,1cyclictest16958-21apt-get22:20:110
225699108,1cyclictest164742sleep319:37:123
225699107,2cyclictest24495-21cat21:00:223
225699107,2cyclictest1220-21sh22:30:023
225699100,9cyclictest19050-21cat20:50:153
225399107,2cyclictest6368-21sort23:30:172
225399107,2cyclictest27379-21expr20:00:132
225399107,2cyclictest25988-21grep21:05:142
225399102,2cyclictest225-21systemd-journal19:33:472
225399100,9cyclictest9416-21taskset19:23:022
225099108,1cyclictest4378-21chrt19:12:271
225099108,1cyclictest16952-21chrt19:40:061
225099107,2cyclictest20465-21chrt20:52:201
225099107,2cyclictest13538-21wc19:30:221
225099101,8cyclictest9070-21cpuspeed_turbos20:30:141
224699108,1cyclictest5790-21expr19:15:150
224699107,2cyclictest3696-21ls19:10:210
224699107,2cyclictest20914-21taskset20:55:070
224699107,2cyclictest17166-21ls20:45:190
224699107,2cyclictest16032-21uniq23:55:160
224699107,2cyclictest14962-21expr19:35:140
224699107,2cyclictest12472-21tr19:30:150
224699103,3cyclictest225-21systemd-journal21:05:000
22569997,2cyclictest22263-21sendmail_mailqu20:55:213
22569997,1cyclictest27261-21expr20:00:133
22539997,1cyclictest23959-21taskset19:53:522
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional