You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 09:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 16, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1648429077,9sleep00-21swapper/019:05:050
1652528460,23sleep10-21swapper/119:05:141
1844828159,6sleep30-21swapper/319:07:493
1828728162,9sleep20-21swapper/219:05:432
1871399263,19cyclictest27054-21unixbench_singl00:10:253
1871399260,14cyclictest0-21swapper/323:14:253
1871399250,5cyclictest141rcu_preempt00:08:433
18704992513,8cyclictest225-21systemd-journal20:35:000
1870799240,22cyclictest0-21swapper/123:17:251
1870499240,21cyclictest0-21swapper/022:19:340
18713992316,5cyclictest55650irq/127-enp2s0-22:28:303
1871399231,4cyclictest3319-21cron23:00:003
1871199230,3cyclictest0-21swapper/223:31:042
18704992321,1cyclictest0-21swapper/022:04:560
1870499230,1cyclictest0-21swapper/021:11:300
1871399225,16cyclictest0-21swapper/300:25:423
1871399222,3cyclictest12930-21turbostat20:05:003
1871399220,4cyclictest0-21swapper/322:37:463
1871199220,19cyclictest0-21swapper/223:58:192
1870499228,13cyclictest13-21ksoftirqd/022:29:410
1871399218,6cyclictest21024-21sh22:50:213
1871399214,16cyclictest15975-21latency_hist22:10:003
18711992120,1cyclictest0-21swapper/222:04:132
1871399204,1cyclictest11494-21diskmemload21:10:003
1871399201,3cyclictest0-21swapper/300:03:453
1871399200,4cyclictest0-21swapper/323:31:103
1871399200,2cyclictest0-21swapper/300:24:113
1871199202,18cyclictest0-21swapper/221:47:502
1871199202,17cyclictest0-21swapper/221:32:472
1870799202,14cyclictest6825-21sensors23:20:231
1870799201,15cyclictest0-21swapper/120:40:121
1870499200,2cyclictest0-21swapper/019:40:130
1870499200,1cyclictest0-21swapper/023:12:440
1870499200,1cyclictest0-21swapper/022:53:050
1871399198,7cyclictest26998-21sh21:37:483
1871399193,16cyclictest0-21swapper/323:54:243
1871399190,4cyclictest0-21swapper/323:55:183
1871399190,3cyclictest0-21swapper/323:25:413
18707991911,5cyclictest0-21swapper/122:06:501
1870799190,11cyclictest13261-21/usr/sbin/munin21:10:201
1870499190,14cyclictest0-21swapper/023:47:450
1871399182,8cyclictest11924-21/usr/sbin/munin21:10:193
1871399181,3cyclictest0-21swapper/321:43:373
1871399180,18cyclictest0-21swapper/321:58:533
18711991817,1cyclictest0-21swapper/200:00:552
1871199180,18cyclictest0-21swapper/200:16:412
1871399173,2cyclictest27259-21idleruntime-cro20:35:003
1871399172,4cyclictest14539-21latency_hist23:45:003
1871399171,13cyclictest1142-21runrttasks00:33:253
1871399170,3cyclictest0-21swapper/323:37:473
1871199171,16cyclictest0-21swapper/222:20:582
1871199170,15cyclictest0-21swapper/223:54:562
1871199170,15cyclictest0-21swapper/221:38:392
1870499174,8cyclictest455-21dbus-daemon00:09:230
304082160,0chrt0-21swapper/222:55:472
1871399166,1cyclictest1205-21tr22:20:133
1871399165,1cyclictest11494-21diskmemload23:48:583
1871399163,4cyclictest55550irq/126-enp2s0-23:15:123
1871399163,2cyclictest0-21swapper/322:30:023
1871399163,1cyclictest11494-21diskmemload23:07:593
1871399163,1cyclictest0-21swapper/320:25:133
1871399162,1cyclictest1515-21grep22:00:253
1871399160,6cyclictest13252-21ssh21:49:113
1871399160,6cyclictest0-21swapper/322:41:263
1871199166,9cyclictest32-21ksoftirqd/222:17:552
1871199161,15cyclictest0-21swapper/221:26:382
1871199160,4cyclictest0-21swapper/219:40:132
1871199160,16cyclictest0-21swapper/222:43:572
18707991614,2cyclictest0-21swapper/121:46:141
1870499161,8cyclictest5994-21basename23:40:000
1871399158,6cyclictest6727-21munin-plugin-st21:00:013
1871399158,6cyclictest31845-21sh21:21:083
1871399158,6cyclictest31457-21ls20:40:243
1871399158,6cyclictest21082-21sh21:34:483
1871399158,6cyclictest17090-21ssh21:50:363
1871399158,6cyclictest14529-21basename20:05:223
1871399158,6cyclictest13162-21sh22:45:313
1871399156,1cyclictest2443-21apache_processe00:35:123
1871399155,9cyclictest8232-21grep21:00:223
1871399152,4cyclictest29914-21python19:30:243
1870799157,3cyclictest11494-21diskmemload00:14:141
1870799154,5cyclictest2115-21latency_hist22:40:001
18707991512,3cyclictest18697-21cyclictest00:23:291
1870799150,0cyclictest0-21swapper/122:50:401
18704991514,1cyclictest0-21swapper/000:24:250
18704991512,2cyclictest94-21kswapd021:40:100
1870499150,1cyclictest0-21swapper/022:06:120
1871399148,5cyclictest7977-21chrt23:21:013
1871399148,5cyclictest6534-21ssh00:18:043
1871399148,5cyclictest4753-21sh21:25:053
1871399148,5cyclictest18550-21tune2fs20:15:163
1871399147,6cyclictest31741-21sh22:19:103
1871399147,6cyclictest22801-21rm22:13:503
1871399147,6cyclictest21599-21grep19:15:133
1871399147,6cyclictest21370-21basename21:15:173
1871399141,2cyclictest0-21swapper/320:50:133
1871399140,13cyclictest7931-21ssh23:02:023
1871199149,1cyclictest141rcu_preempt22:27:592
1871199141,11cyclictest0-21swapper/219:30:132
1871199140,14cyclictest0-21swapper/221:16:352
18707991410,3cyclictest17629-21cat22:30:001
18707991410,2cyclictest9132-21wc00:00:171
18707991410,2cyclictest15123-21tr21:50:121
1870799140,13cyclictest0-21swapper/123:41:121
1870799140,12cyclictest1175-21cut22:20:131
1870499146,2cyclictest23451-21perf19:20:010
18704991412,0cyclictest0-21swapper/021:34:170
1870499140,3cyclictest0-21swapper/000:18:030
1870499140,2cyclictest0-21swapper/021:22:420
1870499140,10cyclictest0-21swapper/000:30:220
178642140,1sleep10-21swapper/123:08:171
1871399137,5cyclictest3612-21expr19:45:153
1871399137,5cyclictest32606-21expr20:45:133
1871399137,5cyclictest27824-21cat20:35:143
1871399137,5cyclictest24186-21expr19:20:133
1871399137,5cyclictest19131-21wc19:10:153
1871399136,2cyclictest20647-21cat20:20:153
1871399134,2cyclictest6856-21cat19:50:173
1871399131,2cyclictest225-21systemd-journal20:00:003
18713991310,2cyclictest31663-21grep19:35:173
18711991312,1cyclictest0-21swapper/223:00:152
18711991312,1cyclictest0-21swapper/222:07:492
1871199131,1cyclictest141rcu_preempt22:39:132
18711991310,2cyclictest31153-21grep23:35:142
18711991310,2cyclictest16656-21grep20:10:192
18711991310,2cyclictest16648-21cat21:50:232
1871199130,3cyclictest11494-21diskmemload00:20:242
1871199130,13cyclictest0-21swapper/222:53:482
1871199130,12cyclictest11498-21apt21:10:002
1871199130,12cyclictest0-21swapper/223:45:062
1870799139,3cyclictest8658-21tr19:55:141
1870799139,3cyclictest22104-21tr21:35:141
1870799139,3cyclictest20567-21ssh23:10:101
1870799139,3cyclictest16640-21tr00:05:131
1870799138,4cyclictest12991-21ls20:05:011
1870799137,5cyclictest25643-21chrt00:29:481
1870799133,5cyclictest17662-21sh20:15:011
18707991310,2cyclictest10092-21grep21:05:171
1870799130,12cyclictest0-21swapper/123:37:181
18704991310,2cyclictest23785-21apache223:31:070
1870499130,7cyclictest0-21swapper/022:35:130
1870499130,6cyclictest0-21swapper/022:20:170
1870499130,4cyclictest0-21swapper/021:39:560
1870499130,1cyclictest23681-21/usr/sbin/munin19:20:110
1871399126,1cyclictest26580-21expr19:25:153
1871199129,2cyclictest4881-21tr20:55:152
1871199129,2cyclictest32015-21sh20:45:002
1871199129,2cyclictest26642-21sh00:10:212
1871199129,2cyclictest26122-21cron22:35:002
1871199129,2cyclictest20532-21chrt00:07:042
1871199129,2cyclictest18328-21grep00:25:142
1871199129,2cyclictest14338-21grep23:25:152
1871199129,2cyclictest13468-21tr20:05:142
1871199129,2cyclictest13444-21grep23:05:192
1871199121,3cyclictest0-21swapper/223:10:202
1871199121,1cyclictest0-21swapper/221:12:012
18711991211,1cyclictest32-21ksoftirqd/223:15:312
18711991210,1cyclictest21124-21head20:20:162
1870799129,2cyclictest4941-21grep19:45:261
1870799129,2cyclictest29986-21grep23:35:001
1870799129,2cyclictest23467-21sed20:25:171
1870799129,2cyclictest20037-21grep19:10:191
1870799129,2cyclictest14406-21tr20:05:211
1870799129,2cyclictest13370-21ssh23:25:021
1870799128,3cyclictest26809-21sh21:56:331
1870799126,6cyclictest38150irq/131-snd_hda23:55:251
1870799125,6cyclictest24841-21/usr/sbin/munin22:15:171
1870799122,3cyclictest29738-21/usr/sbin/munin21:20:171
1870799121,10cyclictest11494-21diskmemload00:40:001
1870799120,1cyclictest0-21swapper/120:50:131
1870799120,1cyclictest0-21swapper/100:15:541
1870799120,11cyclictest0-21swapper/122:40:151
1870499129,2cyclictest795-21tr23:55:210
1870499129,2cyclictest6074-21sed23:20:170
1870499129,2cyclictest26857-21tail19:25:180
1870499129,2cyclictest25146-21wc21:55:260
1870499129,2cyclictest20085-21cron23:09:590
1870499129,2cyclictest19451-21wc20:15:260
1870499128,3cyclictest7378-21ssh00:37:420
1870499124,7cyclictest6760-21sendmail-msp21:00:000
1870499122,2cyclictest0-21swapper/022:10:210
18704991210,2cyclictest25388-21head23:50:270
1870499120,8cyclictest0-21swapper/021:16:060
1870499120,11cyclictest21786-21ssh00:26:460
1870499120,10cyclictest0-21swapper/022:44:080
1871399118,2cyclictest965-21tr19:40:133
1871399110,5cyclictest17678-21basename20:15:003
1871199119,1cyclictest9377-21wc19:55:182
1871199119,1cyclictest8555-21cat21:00:262
1871199119,1cyclictest810-21cat19:40:012
1871199119,1cyclictest7734-21ssh23:40:182
1871199119,1cyclictest4896-21tail19:45:272
1871199119,1cyclictest4249-21wc00:35:272
1871199119,1cyclictest32315-21grep21:40:252
1871199119,1cyclictest29791-21grep21:20:182
1871199119,1cyclictest27850-21sed19:25:272
1871199119,1cyclictest23445-21head20:25:172
1871199119,1cyclictest17354-21ssh22:10:182
1871199119,1cyclictest10873-21ssh22:45:062
1871199118,2cyclictest5735-21sort19:50:002
1871199118,2cyclictest306-21df20:45:152
1871199118,2cyclictest28508-21ssh00:30:292
1871199118,2cyclictest27590-21tr20:35:142
1871199115,5cyclictest0-21swapper/223:21:292
1871199114,5cyclictest23298-21apache_accesses21:55:122
1870799119,1cyclictest23637-21tr23:50:141
1870799118,2cyclictest6883-21grep21:25:261
1870799118,2cyclictest4596-21cut23:00:171
1870799118,2cyclictest31179-21head21:40:151
1870799118,2cyclictest30914-21grep19:35:131
1870799118,2cyclictest30134-21ssh22:55:351
1870799118,2cyclictest29955-21chrt00:31:351
1870799118,2cyclictest29231-21grep19:30:161
1870799118,2cyclictest27952-21ssh21:19:381
1870799118,2cyclictest24021-21tr19:20:131
1870799118,2cyclictest2142-21ssh22:00:511
1870799118,2cyclictest20295-21taskset20:20:031
1870799118,2cyclictest19002-21sort20:15:191
1870799118,2cyclictest18637-21ssh22:10:361
1870799118,2cyclictest1568-21sed20:45:251
1870799118,2cyclictest13193-21ssh21:30:041
1870799118,2cyclictest12911-21sed22:45:251
1870799117,3cyclictest25248-21grep20:30:141
1870799117,3cyclictest14981-21cut23:45:121
1870799112,5cyclictest225-21systemd-journal19:30:011
1870799111,8cyclictest6542-21gltestperf19:50:151
1870799110,10cyclictest0-21swapper/122:33:371
1870499118,2cyclictest7026-21sh23:40:130
1870499118,2cyclictest6255-21ntp_states21:25:210
1870499118,2cyclictest6206-21sh21:45:040
1870499118,2cyclictest5941-21cut19:50:120
1870499118,2cyclictest5240-21chrt19:48:440
1870499118,2cyclictest3896-21ssh23:00:140
1870499118,2cyclictest309882sleep023:15:260
1870499118,2cyclictest30484-21chrt19:33:180
1870499118,2cyclictest30456-21sed20:40:140
1870499118,2cyclictest28228-21tr22:55:140
1870499118,2cyclictest26278-21sed00:10:180
1870499118,2cyclictest2468-21tr20:50:140
1870499118,2cyclictest21600-21rm22:31:330
1870499118,2cyclictest16816-21sh23:26:200
1870499118,2cyclictest16203-21tr21:50:190
1870499118,2cyclictest15400-21ssh22:47:150
1870499118,2cyclictest14120-21sed20:05:180
1870499118,2cyclictest12713-21ssh00:02:320
1870499118,2cyclictest11103-21grep21:05:280
1870499118,2cyclictest10913-21cut20:00:140
1871199108,2cyclictest3417-21open_inodes20:50:202
1871199108,1cyclictest25420-21grep20:30:142
1871199108,1cyclictest22530-21grep19:15:212
1871199108,1cyclictest11925-21nfs4_client20:00:192
1871199103,2cyclictest477-21rs:main2
1871199100,4cyclictest20071-21perf20:20:002
1870799108,1cyclictest7655-21cut21:00:161
1870799108,1cyclictest2384-21cat19:40:251
1870799107,2cyclictest5571-21wc20:55:191
1870799107,2cyclictest21630-21wc19:15:141
1870799103,5cyclictest27622-21grep20:35:151
1870499108,1cyclictest4042sleep019:38:230
1870499108,1cyclictest20988-21sort20:20:160
1870499107,2cyclictest28343-21sort20:35:180
1870499107,2cyclictest24320-21sed20:25:260
1870499107,2cyclictest16622-21sed20:10:190
1870499100,10cyclictest9412-21sed19:55:180
187119998,1cyclictest20358-21ls19:10:242
187049997,1cyclictest16812sleep020:46:220
187049996,2cyclictest7001-21apache_accesses21:00:130
187049993,4cyclictest18904-21/usr/sbin/munin19:10:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional