You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 08:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri May 15, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3018929985,9sleep00-21swapper/019:05:530
3039829367,9sleep20-21swapper/219:08:372
2919329174,12sleep10-21swapper/119:05:191
3017728370,9sleep30-21swapper/319:05:453
122882500,0sleep20-21swapper/223:56:382
3059699280,28cyclictest0-21swapper/223:19:582
3059699280,28cyclictest0-21swapper/222:54:392
3059399251,21cyclictest23603-21diskmemload22:34:171
3059699240,21cyclictest0-21swapper/222:10:352
3059699240,1cyclictest0-21swapper/221:10:502
3060099230,1cyclictest0-21swapper/321:31:053
66592220,1sleep30-21swapper/300:31:593
3060099220,21cyclictest0-21swapper/323:20:493
3060099220,1cyclictest0-21swapper/323:45:423
3060099220,0cyclictest0-21swapper/323:07:423
3060099210,0cyclictest0-21swapper/322:26:343
3059699210,20cyclictest0-21swapper/222:07:512
3059099210,2cyclictest0-21swapper/021:20:390
3059099210,20cyclictest0-21swapper/000:10:130
3059699200,19cyclictest0-21swapper/222:57:572
3059399200,17cyclictest0-21swapper/122:08:441
3059099200,20cyclictest0-21swapper/023:33:010
3059699190,19cyclictest0-21swapper/223:03:232
3059699190,18cyclictest0-21swapper/223:41:332
3059399190,1cyclictest0-21swapper/122:36:121
3059099195,7cyclictest31991-21perf21:15:000
3060099181,17cyclictest0-21swapper/321:47:283
3060099180,15cyclictest0-21swapper/322:06:213
3060099180,0cyclictest0-21swapper/322:42:543
3060099180,0cyclictest0-21swapper/322:39:523
30593991817,1cyclictest0-21swapper/100:15:211
30593991811,7cyclictest38150irq/131-snd_hda22:44:261
3059399180,18cyclictest0-21swapper/100:00:511
30593991714,3cyclictest25-21ksoftirqd/122:57:411
30600991611,3cyclictest55650irq/127-enp2s0-23:42:173
3060099160,3cyclictest0-21swapper/323:14:213
3059699160,12cyclictest0-21swapper/222:44:142
30593991614,1cyclictest25336-21missed_timers21:10:201
3059399160,16cyclictest0-21swapper/122:49:531
3059099162,7cyclictest28737-21munin-run22:50:000
30590991611,3cyclictest906-21aten2_r7power_v19:15:130
3059099160,12cyclictest0-21swapper/021:36:230
3060099153,12cyclictest0-21swapper/322:11:453
30600991513,1cyclictest11986-21cstates22:20:143
30596991512,2cyclictest94-21kswapd023:35:002
3059699150,5cyclictest0-21swapper/200:11:192
3059699150,15cyclictest0-21swapper/200:39:582
3059699150,15cyclictest0-21swapper/200:27:092
3059699150,12cyclictest0-21swapper/221:38:112
3059399155,9cyclictest3092-21apt-get22:15:121
3059399150,0cyclictest0-21swapper/123:53:481
3059099153,1cyclictest141rcu_preempt21:28:110
3059099150,0cyclictest0-21swapper/000:20:130
30600991410,3cyclictest6308-21sed20:30:253
30600991410,2cyclictest17910-21cat21:25:203
3060099140,13cyclictest2330-21cut21:55:133
30596991413,1cyclictest55650irq/127-enp2s0-00:16:202
3059699140,10cyclictest0-21swapper/219:45:132
30593991414,0cyclictest0-21swapper/122:28:131
30593991411,2cyclictest3896-21apache222:53:271
30593991410,3cyclictest29627-21grep20:15:141
3059399140,14cyclictest0-21swapper/100:39:411
3059399140,0cyclictest0-21swapper/123:08:081
3059099149,3cyclictest3612-21grep20:25:250
30590991413,1cyclictest0-21swapper/021:47:110
30590991410,3cyclictest30664-21wc20:15:220
30590991410,3cyclictest11803-21grep00:15:270
30590991410,2cyclictest7268-21grep20:35:140
3059099140,14cyclictest0-21swapper/000:02:380
134192140,0sleep00-21swapper/021:42:050
3060099132,5cyclictest0-21swapper/321:05:123
30600991310,2cyclictest7261-21sed19:25:253
30600991310,2cyclictest2383-21grep21:35:243
3060099130,11cyclictest110612sleep300:35:143
3059699139,3cyclictest3563-21apt22:34:592
3059699139,3cyclictest26358-21head20:05:272
3059699139,3cyclictest17861-21grep00:00:162
3059699139,3cyclictest0-21swapper/222:36:182
3059699139,2cyclictest470-21cron19:15:002
3059699138,1cyclictest0-21swapper/223:45:402
3059699131,7cyclictest0-21swapper/220:15:132
30596991310,2cyclictest9675-21grep23:35:282
30596991310,2cyclictest27756-21head20:10:162
30596991310,2cyclictest21151-21sh22:25:202
30596991310,2cyclictest19114-21sed19:50:262
30596991310,2cyclictest10322-21tail20:40:162
3059399139,3cyclictest19320-21cron22:24:591
3059399139,2cyclictest0-21swapper/119:45:121
3059399133,1cyclictest0-21swapper/123:57:291
3059399132,1cyclictest23603-21diskmemload23:26:541
3059399131,8cyclictest0-21swapper/120:10:121
3059399131,6cyclictest0-21swapper/121:40:131
3059399131,2cyclictest31726-21cupsd00:08:401
30593991311,2cyclictest38150irq/131-snd_hda23:00:151
30593991311,1cyclictest0-21swapper/100:12:021
30593991310,2cyclictest27101-21cut21:50:181
30593991310,2cyclictest25719-21sed21:50:001
30593991310,2cyclictest24749-21grep20:05:161
3059399130,9cyclictest0-21swapper/123:40:131
3059399130,12cyclictest6254-21grep20:30:251
3059099136,6cyclictest29769-21apt-get23:10:000
3059099133,9cyclictest10540-21perf00:35:000
3059099131,6cyclictest13066-21sendmail23:00:000
30590991310,2cyclictest22911-21grep20:00:150
30590991310,2cyclictest19881-21grep22:25:120
3060099129,2cyclictest600-21grep20:20:193
3060099129,2cyclictest5271-21ssh22:55:143
3060099129,2cyclictest4463-21seq22:54:493
3060099129,2cyclictest29509-21ls22:30:213
3060099129,2cyclictest2886-21cron19:20:013
3060099129,2cyclictest27000-21tr00:25:153
3060099129,2cyclictest2242-21grep00:10:173
3060099129,2cyclictest11615-21wc19:35:213
3060099129,2cyclictest10611-21ntpq21:40:203
3060099128,3cyclictest30797-21chrt20:15:243
3060099124,4cyclictest225-21systemd-journal20:00:003
3059699129,2cyclictest9298-21cron20:40:002
3059699129,2cyclictest3888-21apache221:20:102
3059699129,2cyclictest2661-21ssh00:30:112
3059699129,2cyclictest22292-21tr22:45:242
3059699129,2cyclictest2212-21tr20:25:132
3059699129,2cyclictest16041-21grep20:50:262
3059699128,3cyclictest7027-21wc19:25:242
3059699128,3cyclictest18739-21expr00:20:152
3059699128,1cyclictest5304-21chrt23:52:422
3059699122,4cyclictest0-21swapper/200:07:232
3059699120,11cyclictest10908-21ssh21:40:222
3059399129,2cyclictest8815-21grep20:35:271
3059399129,2cyclictest7722-21turbostat.cron19:30:011
3059399129,2cyclictest6629-21egrep23:15:151
3059399129,2cyclictest32527-21grep20:20:151
3059399129,2cyclictest24514-21cron21:30:011
3059399129,2cyclictest21480-21sed19:55:261
3059399129,2cyclictest16008-21cat23:20:191
3059399129,2cyclictest12455-21cron19:40:001
3059399129,1cyclictest21064-21sh00:21:011
3059399125,3cyclictest0-21swapper/120:25:131
3059099129,2cyclictest9598-21grep19:30:250
3059099129,2cyclictest5901-21tr19:25:130
3059099129,2cyclictest5360-21sleep19:25:010
3059099129,2cyclictest3539-21ls21:55:200
3059099129,2cyclictest27251-21grep20:10:130
3059099129,2cyclictest17274-21tail23:40:190
3059099129,2cyclictest14792-21cat23:00:220
3059099129,2cyclictest14447-21grep19:40:260
3059099129,2cyclictest12319-21grep22:20:150
3059099128,2cyclictest24587-21ssh23:25:230
3059099122,1cyclictest141rcu_preempt22:40:180
3059099121,8cyclictest17530-21aten2_r7power_c19:50:130
3059099120,12cyclictest0-21swapper/023:24:300
3059099120,12cyclictest0-21swapper/022:18:150
3059099120,11cyclictest26211-21head00:05:190
273212120,0sleep20-21swapper/223:08:062
3060099119,2cyclictest94-21kswapd023:58:073
3060099119,1cyclictest15566-21sort23:39:593
3060099118,2cyclictest9363-21hostname21:20:173
3060099118,2cyclictest8504-21ssh23:15:283
3060099118,2cyclictest339-21ssh23:50:133
3060099118,2cyclictest32659-21cut21:15:173
3060099118,2cyclictest31053-21seq21:52:383
3060099118,2cyclictest3064-21grep20:25:193
3060099118,2cyclictest2940-21ssh22:15:063
3060099118,2cyclictest27729-21ssh00:05:433
3060099118,2cyclictest26393-21ssh21:10:403
3060099118,2cyclictest26065-21wc20:05:263
3060099118,2cyclictest20614-21ssh22:45:033
3060099118,2cyclictest20488-21ssh00:20:363
3060099118,2cyclictest20294-21ssh00:01:263
3060099118,2cyclictest17644-21egrep19:50:153
3060099118,2cyclictest16299-21taskset20:53:173
3060099118,2cyclictest15146-21fgrep23:00:263
3060099118,2cyclictest14676-21taskset19:42:493
3060099118,2cyclictest12718-21sed20:45:173
3060099118,2cyclictest11637-21cat00:15:263
3060099118,2cyclictest10691-21ssh22:00:143
3060099117,3cyclictest31844-21tr23:30:163
3060099114,3cyclictest225-21systemd-journal23:30:003
3060099113,4cyclictest225-21systemd-journal20:35:013
3060099113,4cyclictest225-21systemd-journal19:50:003
3059699119,1cyclictest32423-21sort20:20:142
3059699118,2cyclictest5871-21wc20:30:212
3059699118,2cyclictest3287-21ssh21:16:252
3059699118,2cyclictest29040-21ssh21:51:042
3059699118,2cyclictest2890-21sh19:20:012
3059699118,2cyclictest26398-21ssh23:26:242
3059699118,2cyclictest21892-21ssh21:47:112
3059699118,2cyclictest21550-21cut21:05:122
3059699118,2cyclictest19731-21uniq21:00:172
3059699118,2cyclictest17193-21ssh23:20:442
3059699118,2cyclictest17189-21sh22:23:202
3059699118,2cyclictest16972-21expr20:55:132
3059699118,2cyclictest1419-21ssh23:11:302
3059699118,2cyclictest12711-21sed20:45:162
3059699118,2cyclictest10619-21grep22:00:132
3059699111,5cyclictest0-21swapper/221:30:342
3059699111,2cyclictest225-21systemd-journal20:05:002
3059699110,11cyclictest0-21swapper/221:25:142
3059399118,2cyclictest9675-21sed19:30:261
3059399118,2cyclictest8642-21grep21:20:141
3059399118,2cyclictest8005-21tr23:35:161
3059399118,2cyclictest3921-21cut19:20:151
3059399118,2cyclictest32591-21ssh21:15:161
3059399118,2cyclictest32377-21cut23:30:171
3059399118,2cyclictest31231-21ssh23:10:201
3059399118,2cyclictest28894-21sh00:25:371
3059399118,2cyclictest27244-21ssh22:10:121
3059399118,2cyclictest27242-21ssh21:30:361
3059399118,2cyclictest25928-21awk23:45:231
3059399118,2cyclictest24287-21chrt20:04:001
3059399118,2cyclictest22080-21cut21:05:161
3059399118,2cyclictest2083-21ssh21:55:101
3059399118,2cyclictest19345-21grep21:00:151
3059399118,2cyclictest17568-21tr19:50:141
3059399118,2cyclictest17148-21ssh22:04:011
3059399118,2cyclictest12720-21cat19:40:121
3059399113,8cyclictest38150irq/131-snd_hda21:35:551
3059099119,2cyclictest3888-21apache223:36:480
3059099119,1cyclictest12746-21sed20:45:160
3059099118,2cyclictest8959-21ssh23:15:500
3059099118,2cyclictest8801-21ssh23:55:130
3059099118,2cyclictest820-21grep21:15:190
3059099118,2cyclictest7994-21ssh22:36:540
3059099118,2cyclictest409-21ssh22:51:430
3059099118,2cyclictest30936-21ssh22:11:440
3059099118,2cyclictest29462-21ssh00:26:040
3059099118,2cyclictest29289-21ssh21:51:160
3059099118,2cyclictest25596-21sort23:45:200
3059099118,2cyclictest24009-21ssh22:07:510
3059099118,2cyclictest21696-21wc21:05:140
3059099118,2cyclictest1941-21ssh22:33:440
3059099118,2cyclictest18020-21gzip20:55:210
3059099118,2cyclictest1590-21grep23:50:210
3059099118,2cyclictest12509-21grep00:35:250
3059099118,2cyclictest10742-21ntp_states20:40:210
3059099118,2cyclictest10445-21sh22:00:080
30590991110,1cyclictest13-21ksoftirqd/023:14:200
3060099108,1cyclictest9887-21taskset19:32:303
3060099108,1cyclictest19066-21taskset21:00:063
3060099107,2cyclictest50742sleep319:21:393
3060099107,2cyclictest31519-21perl19:10:163
3060099107,2cyclictest26973-21taskset20:10:013
3060099107,2cyclictest17806-21sed20:55:193
3060099107,2cyclictest11459-21chrt20:42:053
3060099100,9cyclictest22573-21cat20:00:143
3059699108,1cyclictest8318-21cat19:30:142
3059699108,1cyclictest4480-21ssh22:15:202
3059699108,1cyclictest12804-21aten2_r7power_e19:40:142
3059699108,1cyclictest123032sleep219:38:052
3059699107,2cyclictest4375-21ntpq19:20:192
3059699107,2cyclictest2290-21grep21:55:132
3059699107,2cyclictest21561-21head19:55:252
3059399108,1cyclictest95852sleep120:40:111
3059399107,2cyclictest841-21cut19:15:131
3059399107,2cyclictest30948-21aten2_r7power_v19:10:131
3059399107,2cyclictest17798-21tail20:55:191
3059399107,2cyclictest13326-21readlink20:45:221
3059399106,2cyclictest14576-21tr20:50:141
3059399102,4cyclictest0-21swapper/100:30:131
3059099108,1cyclictest26436-21chrt20:06:050
3059099108,1cyclictest219552sleep019:59:510
3059099108,1cyclictest17146-21taskset19:49:030
3059099108,1cyclictest162232sleep020:52:200
3059099107,2cyclictest25907-21head21:30:170
3059099107,2cyclictest19780-21cut21:00:160
3059099102,2cyclictest0-21swapper/020:30:130
3059099100,9cyclictest31769-21cat20:20:090
3059099100,3cyclictest0-21swapper/019:35:120
305909996,2cyclictest31581-21head19:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional