You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 23:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Aug 26, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2498299622,30cyclictest27-21ksoftirqd/207:15:122
2499499402,22cyclictest35966irq/58-eth010:25:153
2499499401,36cyclictest0-21swapper/310:30:133
2497499407,4cyclictest131rcu_preempt07:20:121
2499499396,21cyclictest35966irq/58-eth007:30:123
2499499392,31cyclictest0-21swapper/310:00:123
2496399386,12cyclictest12-21ksoftirqd/011:30:110
2499499372,23cyclictest35966irq/58-eth011:45:143
2497499373,3cyclictest131rcu_preempt12:30:131
2497499373,3cyclictest131rcu_preempt12:30:131
2499499363,20cyclictest35966irq/58-eth012:00:123
2499499361,32cyclictest0-21swapper/311:40:133
2499499361,30cyclictest0-21swapper/312:35:123
2499499361,24cyclictest0-21swapper/308:10:123
2499499360,32cyclictest0-21swapper/309:55:133
2498299363,22cyclictest27-21ksoftirqd/212:25:122
2497499363,2cyclictest131rcu_preempt07:45:131
24994993524,7cyclictest35966irq/58-eth010:50:133
24994993524,7cyclictest35966irq/58-eth010:50:133
2498299350,27cyclictest27-21ksoftirqd/211:35:132
2499499341,30cyclictest0-21swapper/310:55:123
2499499341,27cyclictest0-21swapper/312:30:133
2499499341,27cyclictest0-21swapper/312:30:133
2499499341,24cyclictest0-21swapper/307:15:123
2496399345,7cyclictest12-21ksoftirqd/009:35:130
2496399341,30cyclictest10651-21diskmemload12:30:130
2496399341,30cyclictest10651-21diskmemload12:30:130
2499499331,28cyclictest0-21swapper/311:15:123
2499499331,28cyclictest0-21swapper/311:15:123
24982993312,18cyclictest261rcuc/210:35:142
2496399337,6cyclictest131rcu_preempt08:15:130
2499499321,24cyclictest0-21swapper/311:00:133
2499499321,23cyclictest0-21swapper/309:35:133
2498299322,22cyclictest9799-21df_inode09:45:162
2496399322,22cyclictest12-21ksoftirqd/009:45:130
2499499311,26cyclictest0-21swapper/311:50:153
24963993126,2cyclictest12-21ksoftirqd/011:29:020
2499499306,15cyclictest35966irq/58-eth011:30:123
2499499301,26cyclictest0-21swapper/308:30:113
2499499301,24cyclictest0-21swapper/309:45:133
2499499301,24cyclictest0-21swapper/309:40:123
2499499301,24cyclictest0-21swapper/309:15:123
2498299302,11cyclictest26308-21ssh09:53:402
2497499302,23cyclictest14193-21ssh11:35:141
2497499301,20cyclictest28692-21kworker/u8:212:25:111
2496399303,24cyclictest32091-21ssh11:45:130
24963993020,7cyclictest12-21ksoftirqd/009:10:120
2499499292,23cyclictest0-21swapper/308:15:123
2499499291,25cyclictest0-21swapper/311:10:123
2499499291,25cyclictest0-21swapper/311:10:123
2498299292,21cyclictest0-21swapper/208:00:132
2498299291,16cyclictest648-21ssh09:40:132
24974992918,2cyclictest21-21ksoftirqd/110:45:151
2496399291,22cyclictest11949-21ssh10:40:120
2496399291,14cyclictest12-21ksoftirqd/007:35:120
2499499281,24cyclictest0-21swapper/312:05:133
2499499281,23cyclictest0-21swapper/312:20:143
2499499281,22cyclictest0-21swapper/307:45:133
2499499281,20cyclictest0-21swapper/310:10:133
2498299283,14cyclictest27-21ksoftirqd/208:40:132
2498299281,9cyclictest1702766ssh10:05:232
2497499281,19cyclictest26304-21rm10:30:131
2496399287,18cyclictest12-21ksoftirqd/007:45:130
2496399286,18cyclictest12-21ksoftirqd/007:30:130
24963992814,4cyclictest12-21ksoftirqd/008:40:160
2496399281,24cyclictest18753-21ssh09:50:130
2499499275,13cyclictest0-21swapper/311:20:143
2499499271,23cyclictest35966irq/58-eth008:00:123
2499499271,22cyclictest0-21swapper/311:35:143
2499499270,19cyclictest0-21swapper/311:25:113
2498299271,9cyclictest2585466ssh12:15:322
2498299271,3cyclictest131rcu_preempt09:35:142
2498299271,15cyclictest26595-21ssh12:00:112
24974992710,3cyclictest21-21ksoftirqd/109:40:041
2496399279,10cyclictest12-21ksoftirqd/011:15:130
2496399279,10cyclictest12-21ksoftirqd/011:15:130
2496399278,5cyclictest131rcu_preempt12:25:110
2496399273,20cyclictest12191-21ssh12:10:140
2496399271,22cyclictest24881-21ssh11:05:140
2496399271,18cyclictest19438-21ssh11:20:130
2499499262,21cyclictest0-21swapper/311:55:133
2499499262,17cyclictest0-21swapper/307:20:123
2499499261,23cyclictest0-21swapper/309:10:113
2499499261,22cyclictest0-21swapper/312:15:123
2499499261,22cyclictest0-21swapper/310:40:013
2499499261,22cyclictest0-21swapper/310:18:163
2499499261,21cyclictest0-21swapper/311:05:123
2499499261,21cyclictest0-21swapper/309:50:123
2499499261,21cyclictest0-21swapper/309:30:113
2499499261,21cyclictest0-21swapper/309:25:133
2499499261,20cyclictest0-21swapper/307:35:113
2499499260,23cyclictest0-21swapper/312:10:123
2498299263,7cyclictest0-21swapper/210:12:162
2498299262,13cyclictest0-21swapper/211:13:552
2498299262,13cyclictest0-21swapper/211:13:552
24982992612,2cyclictest27-21ksoftirqd/211:06:162
2498299261,12cyclictest810366ssh10:00:232
2497499268,12cyclictest30246-21kworker/u8:210:20:131
2497499262,12cyclictest21-21ksoftirqd/109:45:131
2499499251,22cyclictest0-21swapper/310:45:143
2499499251,21cyclictest0-21swapper/310:08:483
2499499251,21cyclictest0-21swapper/309:00:123
2499499251,19cyclictest0-21swapper/308:40:113
2499499251,18cyclictest0-21swapper/307:10:123
2499499251,18cyclictest0-21swapper/307:10:123
24982992512,9cyclictest13357-21ssh11:34:362
2498299251,12cyclictest17618-21ssh11:55:132
2498299250,21cyclictest1861-21timerwakeupswit12:20:312
2497499252,7cyclictest131rcu_preempt09:33:361
2497499251,19cyclictest28475-21rm11:25:121
2496399253,17cyclictest30272-21rm12:20:150
24963992512,11cyclictest12-21ksoftirqd/007:15:140
2496399251,12cyclictest4046-21kthreadcore09:40:220
2499499247,10cyclictest0-21swapper/312:25:113
2499499241,20cyclictest0-21swapper/308:55:143
2499499241,19cyclictest0-21swapper/308:45:113
2499499241,19cyclictest0-21swapper/307:25:113
2499499240,21cyclictest0-21swapper/308:05:123
2498299248,8cyclictest8653-21ssh11:50:152
2498299243,17cyclictest27346irq/40-dwc2_hso11:00:142
2498299242,6cyclictest0-21swapper/209:58:412
2498299242,17cyclictest0-21swapper/209:23:162
2498299241,6cyclictest3151866rm12:37:572
2498299241,6cyclictest2455766rm11:20:482
24982992413,9cyclictest28896-21ssh11:41:382
24982992412,4cyclictest27-21ksoftirqd/209:05:302
24982992412,4cyclictest27-21ksoftirqd/209:05:302
2497499243,7cyclictest131rcu_preempt09:56:161
2497499241,4cyclictest0-21swapper/107:35:191
2496399241,8cyclictest10386-21ntp_states10:55:250
2499499232,17cyclictest0-21swapper/307:55:123
2499499231,19cyclictest0-21swapper/310:40:103
2499499231,19cyclictest0-21swapper/309:05:123
2499499231,19cyclictest0-21swapper/309:05:123
2499499231,19cyclictest0-21swapper/308:20:123
2499499231,19cyclictest0-21swapper/307:40:123
2499499231,18cyclictest0-21swapper/310:20:133
2498299234,13cyclictest131rcu_preempt12:30:222
2498299234,13cyclictest131rcu_preempt12:30:222
2498299233,6cyclictest0-21swapper/209:32:162
2498299232,16cyclictest0-21swapper/210:43:162
2498299231,6cyclictest599766ssh10:54:412
2498299231,6cyclictest599766ssh10:54:412
2498299231,6cyclictest2856066ssh10:48:462
2498299231,6cyclictest2421666ssh10:28:022
24982992312,9cyclictest12741-21ssh10:56:562
2497499237,3cyclictest131rcu_preempt07:40:181
2497499236,3cyclictest131rcu_preempt11:50:231
2497499234,5cyclictest5081-21ssh11:30:111
2497499232,16cyclictest19130-21ssh10:43:161
2496399238,4cyclictest131rcu_preempt08:35:230
2496399236,12cyclictest0-21swapper/007:50:130
2496399234,13cyclictest844-21munin-node10:00:130
2496399233,13cyclictest1393-21cut09:00:150
2496399232,19cyclictest22540-21ssh10:10:120
2496399232,18cyclictest8739-21ls11:50:160
2496399231,19cyclictest9730-21ls09:25:280
2496399231,16cyclictest21040-21ssh10:45:150
24963992311,3cyclictest12-21ksoftirqd/010:50:240
24963992311,3cyclictest12-21ksoftirqd/010:50:240
24963992310,4cyclictest12-21ksoftirqd/009:30:200
24963992310,3cyclictest131rcu_preempt10:15:160
2499499221,18cyclictest0-21swapper/309:20:163
2499499221,18cyclictest0-21swapper/308:50:133
2499499221,18cyclictest0-21swapper/308:35:133
2499499221,18cyclictest0-21swapper/308:25:153
2498299227,12cyclictest0-21swapper/209:15:172
2498299221,7cyclictest1600666ssh09:10:402
2498299221,7cyclictest1371466ssh09:29:072
2498299221,6cyclictest2013666ssh12:14:182
2498299221,12cyclictest26344-21apt-get07:45:142
24982992210,10cyclictest1-21systemd07:43:112
2497499221,15cyclictest1492-21ssh10:50:291
2497499221,15cyclictest1492-21ssh10:50:291
2496399222,18cyclictest23521-21ssh11:03:550
2496399222,18cyclictest19007-21ssh10:06:160
2496399222,15cyclictest1429-21ssh11:10:130
2496399222,15cyclictest1429-21ssh11:10:130
24963992215,3cyclictest12-21ksoftirqd/012:15:210
2496399221,18cyclictest19492-21ls08:45:150
2496399221,17cyclictest6865-21ls07:55:220
24963992210,3cyclictest12-21ksoftirqd/008:55:010
2499499211,17cyclictest0-21swapper/307:50:123
2498299219,9cyclictest984-21sshd08:56:162
2498299217,12cyclictest0-21swapper/211:25:272
2498299217,11cyclictest614-21dbus-daemon08:23:162
2498299212,9cyclictest626-21kthreadcore11:45:202
2498299212,9cyclictest5899-21latency_hist09:05:012
2498299211,9cyclictest11701-21cron07:30:012
24982992111,8cyclictest0-21swapper/208:05:142
2498299211,17cyclictest24663-21sed08:15:202
2498299211,12cyclictest949966ssh12:07:172
2498299211,11cyclictest11435-21kthreadcore11:15:192
2498299211,11cyclictest11435-21kthreadcore11:15:192
2498299211,10cyclictest2633-21cat10:35:032
24982992110,9cyclictest5296-21ssh10:17:002
24982992110,8cyclictest0-21swapper/207:10:162
24982992110,8cyclictest0-21swapper/207:10:162
2497499215,7cyclictest131rcu_preempt10:25:221
2497499215,4cyclictest131rcu_preempt09:00:211
2497499215,3cyclictest131rcu_preempt11:10:211
2497499215,3cyclictest131rcu_preempt11:10:211
2497499214,3cyclictest131rcu_preempt10:05:181
2497499212,14cyclictest28931-21ssh10:12:161
2497499211,3cyclictest131rcu_preempt12:15:261
2497499211,3cyclictest131rcu_preempt12:05:131
2497499211,2cyclictest131rcu_preempt10:58:381
2497499211,17cyclictest32478-21cpuspeed08:25:131
2497499211,17cyclictest29949-21cpuspeed07:15:141
2497499211,17cyclictest28832-21ls11:05:281
2497499211,16cyclictest17608-21ssh11:55:121
2497499211,15cyclictest31652-21ssh10:15:111
2497499210,3cyclictest131rcu_preempt08:35:151
2496399219,3cyclictest12-21ksoftirqd/011:40:260
2496399217,6cyclictest12-21ksoftirqd/010:32:340
2496399214,14cyclictest24699-21ssh12:35:130
2496399213,15cyclictest29168-21ssh09:20:140
2496399212,16cyclictest2805-21kthreadcore08:25:210
2496399212,15cyclictest13705-21apt-get11:35:050
24963992114,4cyclictest0-21swapper/007:20:220
24963992111,6cyclictest0-21swapper/009:55:140
24963992110,3cyclictest12-21ksoftirqd/012:00:140
2498299209,4cyclictest12690-21kthreadcore07:30:182
2498299208,10cyclictest0-21swapper/208:25:272
2498299207,10cyclictest0-21swapper/208:50:192
2498299201,17cyclictest261rcuc/210:20:182
24982992011,7cyclictest0-21swapper/208:35:242
2498299201,15cyclictest4976-21fschecks_time08:30:152
2498299201,11cyclictest5369-21kthreadcore07:20:212
2498299201,10cyclictest18715-21cut08:10:192
24982992010,7cyclictest0-21swapper/207:55:152
2497499209,3cyclictest21-21ksoftirqd/112:01:011
2497499206,2cyclictest131rcu_preempt11:40:281
2497499205,3cyclictest131rcu_preempt11:20:221
2497499205,3cyclictest131rcu_preempt11:00:221
2497499205,3cyclictest131rcu_preempt09:05:171
2497499205,3cyclictest131rcu_preempt09:05:171
2497499205,2cyclictest131rcu_preempt10:35:281
2497499204,3cyclictest131rcu_preempt09:50:281
2497499202,3cyclictest131rcu_preempt07:55:161
2497499201,9cyclictest0-21swapper/109:29:381
2497499201,5cyclictest12303-21diskstats12:10:161
24974992011,6cyclictest0-21swapper/112:20:301
2497499200,3cyclictest0-21swapper/110:00:421
2496399201,6cyclictest24142-21tr07:40:210
2496399201,6cyclictest17843-21sh11:55:150
24963992014,3cyclictest12-21ksoftirqd/009:15:140
24963992014,3cyclictest12-21ksoftirqd/008:05:160
24963992013,4cyclictest12-21ksoftirqd/008:00:170
2496399201,15cyclictest17281-21sh10:25:140
24963992011,4cyclictest12-21ksoftirqd/007:10:120
24963992011,4cyclictest12-21ksoftirqd/007:10:120
24963992011,3cyclictest12-21ksoftirqd/010:35:140
24963992010,7cyclictest11842-21netstat10:20:240
2498299191,9cyclictest19442-21fschecks_time08:45:152
2498299191,15cyclictest16721-21/usr/sbin/munin07:35:252
2498299191,14cyclictest3468-21latency_hist07:55:022
2497499193,3cyclictest131rcu_preempt08:05:021
2497499192,13cyclictest32691-21memory09:20:231
2497499191,6cyclictest24812-21df12:35:151
24974991912,5cyclictest0-21swapper/107:50:181
2497499191,12cyclictest15001-21timerandwakeup09:10:281
24974991910,7cyclictest11931-21apt-get07:30:121
24974991910,6cyclictest28273-21cat09:35:301
2497499190,7cyclictest14971-21kthreadcore08:05:201
2497499190,3cyclictest131rcu_preempt11:45:171
2497499190,3cyclictest0-21swapper/111:19:231
2497499190,3cyclictest0-21swapper/111:19:231
2496399196,3cyclictest12-21ksoftirqd/008:10:180
2496399195,4cyclictest12-21ksoftirqd/012:05:170
2496399191,9cyclictest32277-21latency_hist08:25:020
2499499181,15cyclictest0-21swapper/307:10:023
2498299180,10cyclictest25031-21latency_hist07:10:022
2497499185,6cyclictest930-21snmpd08:20:011
2497499185,2cyclictest131rcu_preempt08:10:221
2497499184,3cyclictest131rcu_preempt08:20:311
2497499181,8cyclictest5010-21grep08:30:161
24974991810,6cyclictest19577-21sshd08:45:161
2497499180,10cyclictest0-21swapper/109:17:361
2496399189,6cyclictest0-21swapper/008:30:240
2496399188,2cyclictest12-21ksoftirqd/009:05:170
2496399188,2cyclictest12-21ksoftirqd/009:05:170
24963991814,2cyclictest12-21ksoftirqd/007:25:200
24963991811,4cyclictest12-21ksoftirqd/008:55:200
2497499179,4cyclictest21-21ksoftirqd/107:10:121
2497499179,4cyclictest21-21ksoftirqd/107:10:121
2497499172,9cyclictest0-21swapper/108:40:131
2497499171,4cyclictest131rcu_preempt08:50:201
24974991711,4cyclictest0-21swapper/108:55:191
24974991710,5cyclictest11695-21cron07:30:011
2496399131,9cyclictest24991-21latency_hist07:10:010
2497499121,8cyclictest24999-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional