You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 19:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Jan 11, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
376699609,8cyclictest131rcu_preempt08:40:110
376699564,38cyclictest17348-21ssh11:50:120
376699562,41cyclictest2887-21ssh12:00:150
378399542,32cyclictest0-21swapper/210:35:132
377599537,7cyclictest131rcu_preempt10:25:151
376699528,10cyclictest131rcu_preempt08:15:130
3788995019,22cyclictest28366irq/58-eth012:10:113
3775994812,4cyclictest131rcu_preempt07:30:121
376699464,35cyclictest14011-21ssh11:30:130
378399453,35cyclictest20693-21ssh09:45:132
376699443,4cyclictest131rcu_preempt07:35:130
376699442,30cyclictest11804-21ssh09:40:120
3766994215,21cyclictest12-21ksoftirqd/009:20:150
376699412,31cyclictest19691-21ssh12:30:110
378399402,26cyclictest17364-21ssh09:25:112
378399401,14cyclictest0-21swapper/207:35:132
376699403,27cyclictest6215-21ssh09:55:140
378399393,16cyclictest1679-21ssh12:20:112
377599393,26cyclictest10781-21ssh11:10:131
3788993833,3cyclictest28366irq/58-eth012:00:133
378399382,23cyclictest8491-21ssh09:20:142
376699382,32cyclictest30891-21ssh10:45:140
378899372,20cyclictest28366irq/58-eth009:00:153
378399372,28cyclictest2097-21rm12:36:472
3775993721,6cyclictest21-21ksoftirqd/110:10:021
3766993727,6cyclictest12-21ksoftirqd/009:35:120
3766993618,8cyclictest12-21ksoftirqd/008:35:120
378899354,17cyclictest28366irq/58-eth010:25:133
3766993524,9cyclictest12-21ksoftirqd/011:00:140
376699352,26cyclictest0-21swapper/010:05:140
3766993521,4cyclictest12-21ksoftirqd/012:35:140
376699351,30cyclictest29646-21ssh09:50:130
3783993411,18cyclictest0-21swapper/212:33:462
377599340,24cyclictest15195-21ssh10:00:121
3788993316,11cyclictest0-21swapper/309:54:583
378899331,23cyclictest28366irq/58-eth011:40:113
378399332,24cyclictest2000-21tr11:40:212
377599333,22cyclictest25325-21ssh11:00:141
3766993324,7cyclictest22662-21diskmemload11:40:120
3788993221,7cyclictest28366irq/58-eth009:30:143
378399322,18cyclictest26327-21ssh11:55:152
378399322,18cyclictest26327-21ssh11:55:142
3775993219,6cyclictest851-21sshd10:55:131
377599321,27cyclictest0-21swapper/112:10:111
377599321,22cyclictest29645-21sh09:50:121
3775993211,13cyclictest21-21ksoftirqd/111:05:131
376699322,23cyclictest629-21ssh10:10:130
376699322,23cyclictest629-21ssh10:10:130
3766993219,9cyclictest0-21swapper/009:15:130
377599312,21cyclictest9244-21apt-get10:15:021
376699313,20cyclictest22951-21ssh11:35:130
3766993113,7cyclictest12-21ksoftirqd/008:10:120
376699311,25cyclictest21988-21sh10:40:110
378899306,18cyclictest28366irq/58-eth010:20:133
378899303,20cyclictest28366irq/58-eth010:50:143
378899303,18cyclictest28366irq/58-eth007:55:113
3783993023,4cyclictest4064-21ssh10:30:112
3783993023,4cyclictest4064-21ssh10:30:112
378399302,11cyclictest8395-21/usr/sbin/munin11:45:212
376699305,18cyclictest9592-21ssh10:15:040
3766993020,8cyclictest0-21swapper/011:25:120
378899291,16cyclictest28366irq/58-eth011:35:123
378399294,17cyclictest22346irq/40-dwc2_hso07:55:112
378399292,20cyclictest6722-21ssh09:35:282
376699292,22cyclictest30410-21/usr/sbin/munin09:30:290
378899282,17cyclictest28366irq/58-eth008:00:143
378899282,17cyclictest28366irq/58-eth008:00:133
378399282,21cyclictest0-21swapper/212:04:462
377599283,4cyclictest131rcu_preempt09:45:141
3775992811,12cyclictest21-21ksoftirqd/109:30:281
3775992810,9cyclictest19726-21ssh11:15:131
377599280,21cyclictest1683-21ssh12:20:101
376699284,16cyclictest12738-21kworker/u8:010:55:130
376699282,19cyclictest23014-21ssh09:26:280
3766992818,4cyclictest0-21swapper/010:20:130
378899276,14cyclictest0-21swapper/309:35:123
378899272,20cyclictest28366irq/58-eth010:15:133
378899272,19cyclictest28366irq/58-eth010:35:123
378399272,20cyclictest0-21swapper/207:10:142
378399272,19cyclictest0-21swapper/208:50:132
378399272,18cyclictest0-21swapper/209:40:132
3775992716,8cyclictest25920-21ssh09:30:001
376699272,19cyclictest15146-21ssh11:47:460
378899268,14cyclictest0-21swapper/310:05:123
378899263,15cyclictest0-21swapper/310:47:453
3788992616,7cyclictest28366irq/58-eth009:25:103
3788992613,10cyclictest28366irq/58-eth011:30:113
378399263,14cyclictest22346irq/40-dwc2_hso08:15:152
378399262,20cyclictest22662-21diskmemload12:06:462
378399262,16cyclictest22346irq/40-dwc2_hso08:05:132
378399261,21cyclictest22662-21diskmemload10:43:462
377599263,14cyclictest10683-21ssh12:25:141
377599262,22cyclictest15719-21ssh09:23:281
376699266,8cyclictest131rcu_preempt10:25:140
378899251,18cyclictest28366irq/58-eth011:25:133
378899251,15cyclictest28366irq/58-eth012:30:123
3788992510,10cyclictest28366irq/58-eth011:15:143
378399253,16cyclictest22346irq/40-dwc2_hso07:20:112
378399252,5cyclictest620-21ssh10:10:132
378399252,5cyclictest620-21ssh10:10:122
378399252,19cyclictest0-21swapper/208:40:142
3783992512,6cyclictest24764-21ssh10:59:532
3783992511,11cyclictest10516-21kthreadcore10:50:232
377599259,11cyclictest21-21ksoftirqd/109:44:561
377599253,4cyclictest21-21ksoftirqd/109:00:141
377599250,12cyclictest134-21systemd-journal10:50:011
376699251,8cyclictest15516-21/usr/sbin/munin12:05:250
378899244,9cyclictest28366irq/58-eth012:20:103
378899242,19cyclictest28366irq/58-eth007:20:123
378399243,6cyclictest0-21swapper/211:36:452
378399243,15cyclictest0-21swapper/207:15:252
378399242,18cyclictest2942-21ssh09:51:352
3783992410,11cyclictest32679-21kthreadcore10:45:222
377599242,17cyclictest28686-21ssh11:20:121
377599241,18cyclictest11821-21ssh12:05:141
3775992410,3cyclictest0-21swapper/110:35:391
376699242,15cyclictest839-21munin-node08:25:130
376699241,7cyclictest5811-21unixbench_multi12:20:270
3766992413,4cyclictest12-21ksoftirqd/007:50:130
378899236,6cyclictest28366irq/58-eth009:23:293
378899235,14cyclictest28366irq/58-eth011:45:113
378899231,16cyclictest0-21swapper/310:40:123
3788992311,4cyclictest28466irq/59-eth012:05:123
378399232,6cyclictest0-21swapper/210:22:442
378399232,16cyclictest0-21swapper/210:28:462
378399231,17cyclictest0-21swapper/209:30:122
378399231,11cyclictest19750-21ssh11:15:132
378399230,17cyclictest0-21swapper/209:00:132
377599231,18cyclictest608-21ls07:40:141
3775992311,2cyclictest21-21ksoftirqd/112:03:241
376699231,17cyclictest20731-21ssh12:10:120
378899222,16cyclictest28366irq/58-eth007:10:133
378899222,10cyclictest28366irq/58-eth011:10:133
3788992217,3cyclictest321rcuc/311:59:003
3788992217,3cyclictest321rcuc/311:58:593
378899221,13cyclictest28366irq/58-eth010:55:123
378399229,10cyclictest17523-21fschecks_count11:50:162
378399223,12cyclictest22346irq/40-dwc2_hso07:50:132
378399222,6cyclictest0-21swapper/209:18:282
378399222,17cyclictest25544-21fschecks_time11:00:172
378399222,16cyclictest24615-21cron08:05:012
378399222,16cyclictest24615-21cron08:05:002
378399222,15cyclictest15062-21ssh10:00:052
378399222,15cyclictest0-21swapper/210:07:342
378399222,15cyclictest0-21swapper/209:13:282
378399222,10cyclictest6331-21kthreadcore07:45:202
3783992215,4cyclictest924-21snmpd08:45:132
3783992215,4cyclictest924-21snmpd08:45:122
3783992212,7cyclictest14707-21cron10:00:012
3783992211,9cyclictest8800-21ssh11:08:042
378399221,18cyclictest13033-21kthreadcore10:15:242
3783992211,8cyclictest10663-21apt-get12:25:142
377599228,3cyclictest21-21ksoftirqd/111:54:501
377599228,10cyclictest6205-21apt-get09:55:151
377599227,10cyclictest21-21ksoftirqd/110:30:121
377599227,10cyclictest21-21ksoftirqd/110:30:111
377599221,8cyclictest201rcuc/107:20:281
3775992212,7cyclictest28038-21sshd11:55:221
3775992212,7cyclictest28038-21sshd11:55:211
377599221,18cyclictest10241-21ls08:20:221
3775992211,3cyclictest21-21ksoftirqd/109:12:141
3775992210,9cyclictest27299-21ssh10:25:011
3775992210,3cyclictest21-21ksoftirqd/110:52:141
3766992211,8cyclictest13345-21cron07:20:020
376699221,17cyclictest13008-21latency_hist09:00:010
376699220,19cyclictest3409-21ls08:50:010
376699220,19cyclictest3409-21ls08:50:010
3788992116,3cyclictest321rcuc/312:27:243
3788992111,6cyclictest0-21swapper/310:10:303
3788992111,6cyclictest0-21swapper/310:10:293
378899211,12cyclictest0-21swapper/309:55:153
378399217,5cyclictest27-21ksoftirqd/211:30:202
378399214,14cyclictest22346irq/40-dwc2_hso11:10:142
378399212,16cyclictest19492-21kthreadcore07:25:212
378399212,15cyclictest0-21swapper/212:19:462
378399211,10cyclictest20931-21fschecks_time12:10:152
378399211,10cyclictest13226-21kthreadcore08:25:212
377599217,5cyclictest21-21ksoftirqd/111:41:091
377599212,14cyclictest26421-21ssh12:32:451
3775992112,6cyclictest0-21swapper/111:45:111
377599211,17cyclictest23483-21sed11:35:181
3775992111,6cyclictest21-21ksoftirqd/111:30:111
377599211,14cyclictest25034-21cut08:05:151
3775992110,8cyclictest19757-21cron08:00:021
376699218,10cyclictest17441-21grep07:20:260
376699214,12cyclictest0-21swapper/010:50:140
3766992114,4cyclictest0-21swapper/012:25:150
3766992111,8cyclictest12-21ksoftirqd/009:45:120
376699211,14cyclictest839-21munin-node07:55:120
3766992110,3cyclictest131rcu_preempt10:30:170
3766992110,3cyclictest131rcu_preempt10:30:160
378899209,6cyclictest0-21swapper/310:32:083
378899209,6cyclictest0-21swapper/310:32:073
378899204,11cyclictest0-21swapper/311:00:143
378899203,12cyclictest28366irq/58-eth009:45:133
3788992015,3cyclictest321rcuc/311:50:023
3788992015,3cyclictest321rcuc/309:10:503
378899201,12cyclictest0-21swapper/307:35:173
378399203,15cyclictest28050-21latency_hist07:35:022
378399202,14cyclictest748-21fschecks_time07:40:162
378399201,9cyclictest30654-21kthreadcore08:10:182
378399201,9cyclictest29147-21ssh11:20:182
378399201,9cyclictest22355-21kthreadcore08:35:192
378399201,9cyclictest18631-21kthreadcore08:30:222
378399201,9cyclictest18631-21kthreadcore08:30:212
378399201,14cyclictest7070-21fschecks_time08:20:142
378399201,10cyclictest9583-21kthreadcore08:55:202
378399201,10cyclictest9100-21timerandwakeup11:25:292
378399201,10cyclictest21630-21find09:05:272
377599209,8cyclictest3537-21irqstats09:35:191
377599209,3cyclictest21-21ksoftirqd/107:25:271
377599209,3cyclictest0-21swapper/111:27:101
377599202,16cyclictest20368-21head08:30:301
377599202,16cyclictest20368-21head08:30:291
3775992014,4cyclictest0-21swapper/109:15:181
3775992011,6cyclictest21-21ksoftirqd/107:10:151
377599200,17cyclictest14951-21latency08:25:221
376699202,12cyclictest1876-21ssh11:05:140
376699201,15cyclictest17931-21ls09:05:030
376699201,12cyclictest8588-21ls07:15:010
376699201,10cyclictest19700-21/usr/sbin/munin11:15:140
3766992010,7cyclictest1458-21date12:20:010
378899193,8cyclictest28366irq/58-eth009:15:143
378899193,11cyclictest0-21swapper/312:38:463
378899193,11cyclictest0-21swapper/309:40:293
378899192,6cyclictest0-21swapper/312:16:463
377599196,5cyclictest21-21ksoftirqd/110:41:331
377599196,3cyclictest0-21swapper/110:10:401
377599196,3cyclictest0-21swapper/110:10:391
377599194,9cyclictest21-21ksoftirqd/107:50:271
3775991912,5cyclictest0-21swapper/108:10:251
3775991911,5cyclictest0-21swapper/108:45:231
3775991911,5cyclictest0-21swapper/108:45:221
377599190,12cyclictest12982-21kworker/1:209:09:371
376699199,2cyclictest131rcu_preempt07:50:000
376699192,2cyclictest131rcu_preempt11:10:130
376699192,13cyclictest14721-21apt-get10:00:030
376699192,11cyclictest0-21swapper/008:50:030
376699191,12cyclictest0-21swapper/008:00:130
376699191,12cyclictest0-21swapper/008:00:120
376699190,16cyclictest708-21cut07:40:170
376699190,14cyclictest26597-21missed_timers09:10:230
378899183,10cyclictest0-21swapper/307:15:243
378899182,10cyclictest0-21swapper/308:50:043
378899182,10cyclictest0-21swapper/308:45:143
378899182,10cyclictest0-21swapper/308:45:133
378899182,10cyclictest0-21swapper/308:15:143
378899182,10cyclictest0-21swapper/308:10:133
378899182,10cyclictest0-21swapper/308:05:133
3788991812,4cyclictest0-21swapper/311:05:393
3788991811,4cyclictest0-21swapper/311:20:223
378899181,12cyclictest0-21swapper/308:55:153
377599188,8cyclictest21-21ksoftirqd/107:37:021
377599188,2cyclictest21-21ksoftirqd/112:38:161
377599183,13cyclictest23872-21unixbench_multi08:00:311
377599183,13cyclictest23872-21unixbench_multi08:00:301
377599181,7cyclictest131rcu_preempt08:35:261
3775991813,3cyclictest0-21swapper/108:15:311
376699182,13cyclictest26330-21kthreadcore07:30:220
376699181,9cyclictest20257-21timerwakeupswit08:30:290
376699181,9cyclictest20257-21timerwakeupswit08:30:280
3766991810,5cyclictest2450-21cron12:00:010
3766991810,5cyclictest2450-21cron12:00:000
378899172,9cyclictest0-21swapper/308:35:043
378899172,10cyclictest0-21swapper/308:30:143
378899172,10cyclictest0-21swapper/308:30:133
3788991712,3cyclictest321rcuc/308:25:233
3788991712,3cyclictest0-21swapper/308:40:273
3788991712,3cyclictest0-21swapper/307:50:233
3788991711,4cyclictest0-21swapper/310:00:443
3788991711,4cyclictest0-21swapper/307:25:203
378899171,13cyclictest0-21swapper/307:30:173
377599175,3cyclictest131rcu_preempt12:16:231
377599173,11cyclictest9208-21irqrtprio07:15:181
3775991712,2cyclictest21-21ksoftirqd/108:55:281
377599170,8cyclictest4695-21tr08:50:201
3766991711,3cyclictest0-21swapper/010:35:130
376699171,12cyclictest32398-21ntp_states11:20:260
376699171,12cyclictest22678-21basename09:10:010
376699171,10cyclictest10628-21timerwakeupswit08:20:270
3788991611,3cyclictest0-21swapper/309:05:133
377599169,4cyclictest21-21ksoftirqd/107:10:011
377599167,5cyclictest5769-21irqcore07:45:171
377599163,10cyclictest29519-21latency08:40:211
376699162,11cyclictest27228-21kthreadcore08:05:210
376699161,12cyclictest22076-21grep07:25:280
3788991510,3cyclictest0-21swapper/308:20:143
3788991510,3cyclictest0-21swapper/307:45:263
3788991510,3cyclictest0-21swapper/307:40:233
376699151,11cyclictest3774-21wc07:10:020
378399121,8cyclictest3665-21latency_hist07:10:022
378899101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional