You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-16 - 23:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 16, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186929573,16sleep00-21swapper/007:05:220
327528472,8sleep30-21swapper/307:07:263
326928471,9sleep20-21swapper/207:07:202
336828265,13sleep10-21swapper/107:08:361
356899280,3cyclictest0-21swapper/209:37:262
356199288,3cyclictest141rcu_preempt09:05:160
356899270,0cyclictest0-21swapper/209:12:202
356199260,26cyclictest0-21swapper/011:02:160
3561992520,1cyclictest13-21ksoftirqd/010:09:310
136092250,0sleep20-21swapper/212:15:022
357199240,23cyclictest0-21swapper/310:07:403
356899240,0cyclictest0-21swapper/210:41:512
356699248,3cyclictest452-21systemd-logind12:03:561
3561992420,3cyclictest13-21ksoftirqd/009:21:500
3561992420,1cyclictest13-21ksoftirqd/011:47:020
356899230,23cyclictest0-21swapper/210:22:322
356899230,23cyclictest0-21swapper/210:22:322
356899230,23cyclictest0-21swapper/210:11:112
356699233,5cyclictest10690-21munin-run07:25:001
3561992319,1cyclictest13-21ksoftirqd/010:28:160
356199230,20cyclictest0-21swapper/010:20:130
356199230,20cyclictest0-21swapper/010:20:130
357199220,22cyclictest21322-21kworker/3:010:18:023
356899220,19cyclictest0-21swapper/210:47:542
356899220,0cyclictest0-21swapper/210:59:272
3561992220,2cyclictest13-21ksoftirqd/009:46:400
3561992217,5cyclictest13-21ksoftirqd/010:55:020
356199220,1cyclictest0-21swapper/009:14:570
356899211,1cyclictest141rcu_preempt09:55:162
356899210,0cyclictest0-21swapper/212:21:572
356699210,1cyclictest0-21swapper/109:35:261
356899200,3cyclictest0-21swapper/210:52:532
356899200,20cyclictest0-21swapper/211:09:392
3561992015,1cyclictest13-21ksoftirqd/010:01:170
356199200,19cyclictest2286-21sh12:26:560
3571991917,1cyclictest39-21ksoftirqd/309:35:323
3568991918,1cyclictest0-21swapper/211:02:162
356899190,2cyclictest0-21swapper/210:33:222
356899190,1cyclictest0-21swapper/212:08:142
356899190,19cyclictest0-21swapper/212:31:472
356199192,3cyclictest141rcu_preempt09:53:440
3561991918,1cyclictest0-21swapper/009:15:460
3571991813,3cyclictest0-21swapper/311:43:083
357199181,17cyclictest27091-21kworker/3:109:43:123
3568991817,1cyclictest0-21swapper/210:19:052
3568991817,1cyclictest0-21swapper/209:32:042
356899180,1cyclictest0-21swapper/211:22:102
356899180,16cyclictest0-21swapper/210:38:282
356699189,2cyclictest28788-21sed11:25:171
356699188,4cyclictest25356-21apache211:52:381
356699187,10cyclictest17993-21cat10:20:261
356699187,10cyclictest17993-21cat10:20:261
356699184,13cyclictest31067-21apache_accesses09:50:101
356699180,3cyclictest0-21swapper/110:11:211
356199181,4cyclictest141rcu_preempt07:55:180
178622180,4sleep10-21swapper/109:22:271
357199171,15cyclictest0-21swapper/312:08:143
356899171,1cyclictest141rcu_preempt09:20:232
356899170,2cyclictest0-21swapper/212:11:162
356899170,0cyclictest141rcu_preempt11:38:042
356699174,5cyclictest225-21systemd-journal11:00:001
356699170,0cyclictest0-21swapper/109:29:011
356199172,1cyclictest5943-21latency_hist07:15:000
3561991711,2cyclictest13-21ksoftirqd/012:18:340
357199162,11cyclictest24962-21awk07:55:003
357199160,13cyclictest0-21swapper/310:00:083
356899162,1cyclictest3911-21basename07:10:172
3568991613,2cyclictest32-21ksoftirqd/211:30:002
3568991613,2cyclictest32-21ksoftirqd/209:10:012
356699169,2cyclictest19100-21wc10:40:261
356699168,2cyclictest6676-21ssh12:30:111
356699168,2cyclictest28729-21ssh09:10:011
356699167,3cyclictest14821-21cron10:00:001
356199168,3cyclictest13-21ksoftirqd/010:40:140
357199150,13cyclictest0-21swapper/311:51:483
356899150,13cyclictest0-21swapper/212:26:492
356699159,2cyclictest27904-21grep09:05:251
356699158,2cyclictest2260-21wc08:15:001
356699155,10cyclictest2014-21ssh11:48:181
356699150,1cyclictest0-21swapper/109:15:291
356699150,15cyclictest0-21swapper/111:44:391
356199157,1cyclictest141rcu_preempt09:29:320
356199156,9cyclictest25356-21apache212:33:440
356199156,6cyclictest28377-21diskmemload12:07:340
356199154,6cyclictest141rcu_preempt11:05:540
356199152,2cyclictest0-21swapper/008:50:270
356199151,4cyclictest141rcu_preempt10:56:560
3561991514,1cyclictest0-21swapper/009:59:490
356199151,1cyclictest141rcu_preempt11:40:590
356199151,0cyclictest141rcu_preempt10:37:180
356199151,0cyclictest141rcu_preempt10:11:080
3571991410,3cyclictest4549-21wc08:20:003
3571991410,3cyclictest26548-21dump-pmu-power11:05:003
3571991410,3cyclictest10079-21ssh10:35:193
356899149,3cyclictest14004-21cron08:40:002
356899147,5cyclictest55650irq/127-enp2s0-11:49:072
356899143,7cyclictest4573-21munin-node09:54:322
356899140,1cyclictest0-21swapper/211:43:512
356899140,1cyclictest0-21swapper/208:45:142
356699147,2cyclictest19696-21grep07:40:251
356699145,4cyclictest13884-21apt-get12:15:141
356199148,1cyclictest13-21ksoftirqd/011:34:040
357199139,3cyclictest7344-21grep12:10:233
357199139,3cyclictest14963-21wc12:15:203
357199131,1cyclictest0-21swapper/309:55:143
357199131,11cyclictest0-21swapper/309:25:573
3571991310,2cyclictest8607-21sed12:30:263
3571991310,2cyclictest785-21tr10:30:153
3571991310,2cyclictest3283-21cron11:30:003
3571991310,2cyclictest25990-21tr10:45:153
3571991310,2cyclictest24621-21chrt12:01:293
3571991310,2cyclictest21171-21grep07:45:173
3571991310,2cyclictest14036-21grep11:35:283
3568991312,1cyclictest0-21swapper/211:54:382
3568991310,2cyclictest3160-21sed08:15:182
3568991310,2cyclictest15934-21grep07:35:152
3568991310,1cyclictest0-21swapper/210:29:562
356899130,11cyclictest0-21swapper/209:27:442
356899130,0cyclictest0-21swapper/211:56:422
356899130,0cyclictest0-21swapper/211:56:422
356899130,0cyclictest0-21swapper/209:41:042
3566991310,2cyclictest23896-21ls09:45:221
3566991310,2cyclictest20954-21df07:45:141
3566991310,2cyclictest17800-21cron07:40:001
356699130,3cyclictest0-21swapper/112:26:521
356699130,0cyclictest0-21swapper/108:55:051
356199139,1cyclictest141rcu_preempt10:33:000
356199133,1cyclictest141rcu_preempt11:50:210
356199131,6cyclictest2130-21awk08:15:000
356199131,1cyclictest141rcu_preempt12:20:120
3561991311,2cyclictest13-21ksoftirqd/011:27:280
3561991311,1cyclictest13-21ksoftirqd/011:24:490
3561991310,2cyclictest5516-21grep08:20:180
3561991310,2cyclictest26276-21ssh10:45:160
3561991310,2cyclictest22857-21sleep12:40:000
356199130,13cyclictest0-21swapper/009:32:460
356199130,12cyclictest13-21ksoftirqd/007:15:130
357199129,2cyclictest6904-21head07:15:173
357199129,2cyclictest4609-21tail11:30:173
357199129,2cyclictest2259-21grep08:15:003
357199129,2cyclictest15841-21grep12:35:183
357199129,2cyclictest15047-21sed08:40:183
357199129,2cyclictest12044-21grep08:35:163
357199128,3cyclictest4069-21tr07:10:163
357199128,3cyclictest13914-21grep09:20:153
357199127,4cyclictest11196-21/usr/sbin/munin11:15:253
357199126,3cyclictest6404-21ssh10:14:123
357199123,5cyclictest225-21systemd-journal08:21:363
357199123,2cyclictest225-21systemd-journal09:50:003
357199121,1cyclictest0-21swapper/312:21:463
356899129,2cyclictest7927-21grep08:25:172
356899129,2cyclictest3107-21cut11:10:142
356899129,2cyclictest10240-21grep08:30:142
3568991212,0cyclictest0-21swapper/209:18:082
3568991210,1cyclictest32158-21turbostat.cron08:10:002
3568991210,1cyclictest12121-21grep11:15:162
356899120,1cyclictest141rcu_preempt12:02:122
356899120,11cyclictest14336-21tr08:40:142
356699129,2cyclictest4487-21cron08:20:001
356699129,2cyclictest26829-21cat10:45:201
356699129,2cyclictest19645-21grep08:50:181
356699129,2cyclictest16878-21cat10:40:001
356699129,2cyclictest12412-21cut11:35:161
356699128,4cyclictest38150irq/131-snd_hda12:24:201
356699128,3cyclictest27422-21cron11:25:001
356699128,3cyclictest27054-21taskset07:55:541
356699128,3cyclictest25134-21fgrep09:00:221
356699126,4cyclictest38150irq/131-snd_hda07:50:121
3566991211,1cyclictest17426-21kworker/1:211:19:471
356699120,11cyclictest0-21swapper/112:10:061
356699120,11cyclictest0-21swapper/111:57:051
356699120,11cyclictest0-21swapper/111:57:051
356199129,2cyclictest4697-21wc11:10:260
356199129,2cyclictest29334-21tail08:00:270
356199129,2cyclictest16445-21grep07:35:180
356199128,3cyclictest13-21ksoftirqd/009:40:270
356199126,4cyclictest13-21ksoftirqd/011:38:260
356199124,7cyclictest19116-21sendmail11:20:000
3561991210,2cyclictest6214-21uniq12:10:140
356199120,3cyclictest13-21ksoftirqd/010:15:490
356199120,11cyclictest11905-21chrt08:35:130
357199118,2cyclictest9790-21ssh10:55:013
357199118,2cyclictest6779-21wc09:15:263
357199118,2cyclictest4524-21ssh11:10:233
357199118,2cyclictest31473-21expr09:50:133
357199118,2cyclictest30608-21ssh12:25:053
357199118,2cyclictest29803-21ssh11:06:013
357199118,2cyclictest29616-21df09:10:153
357199118,2cyclictest27168-21taskset07:57:233
357199118,2cyclictest25667-21taskset09:01:063
357199118,2cyclictest25149-21grep10:25:153
357199118,2cyclictest22541-21tr08:55:203
357199118,2cyclictest20793-21cat11:20:203
357199118,2cyclictest19298-21grep08:50:153
357199118,2cyclictest18851-21wc08:50:003
357199118,2cyclictest17080-21grep10:20:173
357199118,2cyclictest17080-21grep10:20:173
357199118,2cyclictest14095-21cut07:30:173
357199118,2cyclictest13541-21ssh11:55:143
357199118,2cyclictest13541-21ssh11:55:143
357199118,2cyclictest11496-21df07:25:163
357199118,2cyclictest11278-21seq08:30:323
357199117,3cyclictest17958-21ssh10:40:163
357199113,4cyclictest1142-21runrttasks10:50:143
357199111,3cyclictest28816-21ssh11:45:143
3571991111,0cyclictest0-21swapper/309:30:013
357199110,10cyclictest10365-21taskset07:21:053
356899118,2cyclictest815-21cat08:10:182
356899118,2cyclictest6099-21sh11:30:392
356899118,2cyclictest29889-21wc08:05:002
356899118,2cyclictest28568-21ssh09:48:342
356899118,2cyclictest22122-21grep08:55:172
356899118,2cyclictest12037-21grep07:25:202
356899113,4cyclictest225-21systemd-journal09:01:282
356899111,9cyclictest311rcuc/210:02:502
356699118,2cyclictest9113-21ssh11:33:031
356699118,2cyclictest7183-21cut08:25:141
356699118,2cyclictest4259-21rm10:50:571
356699118,2cyclictest28925-21ssh10:27:151
356699118,2cyclictest27583-21rm10:06:561
356699118,2cyclictest2753-21ssh11:10:041
356699118,2cyclictest2378-21ssh09:32:391
356699118,2cyclictest2032-21rm10:30:231
356699118,2cyclictest14894-21sh10:19:221
356699118,2cyclictest10171-21cut08:30:141
356699117,3cyclictest29310-21ssh11:05:371
356699116,2cyclictest15126-21ssh10:00:021
356699111,5cyclictest225-21systemd-journal07:15:001
356699110,7cyclictest0-21swapper/112:35:221
356699110,10cyclictest11166-21wc07:25:151
356699110,10cyclictest0-21swapper/111:01:401
356199118,2cyclictest7909-21ssh09:35:280
356199118,2cyclictest3053-21cut08:15:170
356199118,2cyclictest24649-21chrt07:51:180
356199118,2cyclictest24591-21sed09:00:160
356199118,2cyclictest23307-21chrt08:57:210
356199118,2cyclictest21284-21ssh12:00:120
356199118,2cyclictest21012-21sed07:45:140
356199118,2cyclictest15006-21grep08:40:180
356199118,2cyclictest14632-21wc11:55:240
356199118,2cyclictest14632-21wc11:55:240
357199108,1cyclictest29601-21chrt08:02:513
357199108,1cyclictest26346-21cut09:05:133
357199107,2cyclictest30523-21expr08:05:153
357199107,2cyclictest18194-21wget07:40:143
357199104,4cyclictest15701-21apache_volume07:35:133
356899108,2cyclictest24138-21taskset07:50:192
356899108,2cyclictest20625-21cat08:50:272
356899108,1cyclictest14746-21ssh12:35:012
356899107,2cyclictest9703-21tr07:20:192
356899100,9cyclictest23424-21ssh10:05:032
356899100,1cyclictest0-21swapper/207:30:142
356699109,1cyclictest0-21swapper/109:40:011
356699108,1cyclictest61712sleep108:20:241
356699108,1cyclictest318072sleep108:05:411
356699108,1cyclictest29554-21ssh12:05:101
356699108,1cyclictest14265-21taskset08:40:101
356699108,1cyclictest13526-21cat07:30:151
356699107,2cyclictest28618-21cut08:00:201
356699107,2cyclictest16679-21apt-get08:45:141
356699107,2cyclictest12705-21hostname08:35:191
356199108,1cyclictest13-21ksoftirqd/007:25:170
356199107,2cyclictest9210-21cut07:20:150
356199107,2cyclictest30116-21grep08:05:140
356199107,2cyclictest17775-21grep08:45:200
356199107,2cyclictest13408-21cut07:30:140
356199107,2cyclictest10632-21head08:30:170
356199101,2cyclictest141rcu_preempt08:25:150
35719997,1cyclictest91592sleep308:28:473
35689998,1cyclictest6718-21df_abs07:15:152
35689998,1cyclictest20556-21aten2_r7power_p07:45:132
35689997,2cyclictest18465-21expr07:40:142
35689997,1cyclictest251712sleep207:55:002
35689991,7cyclictest5106-21cstates08:20:162
35669997,1cyclictest6319-21grep07:15:141
35619991,1cyclictest141rcu_preempt07:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional