You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 00:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 07, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1611699651,3cyclictest0-21swapper/207:40:132
1610799634,46cyclictest28321-21ssh11:45:120
1611699622,5cyclictest9321-21apt-get12:10:132
1611699622,5cyclictest9321-21apt-get12:10:132
1611699621,5cyclictest0-21swapper/208:55:132
16124996037,13cyclictest35966irq/58-eth012:20:143
1611699601,48cyclictest0-21swapper/212:30:152
1610799607,41cyclictest11437-21ssh09:15:130
1610799607,41cyclictest11437-21ssh09:15:120
1610799604,8cyclictest131rcu_preempt08:40:140
1610799604,8cyclictest131rcu_preempt08:40:140
1611699581,7cyclictest0-21swapper/207:20:122
1610799576,39cyclictest13652-21ssh12:30:140
1611699532,45cyclictest0-21swapper/207:35:132
1610799502,37cyclictest874-21ssh11:30:130
1612499496,26cyclictest35966irq/58-eth012:05:123
16124994928,12cyclictest35966irq/58-eth009:25:133
1611699481,3cyclictest0-21swapper/208:20:142
1611699470,4cyclictest5029-21apt-get11:50:142
1611099471,4cyclictest131rcu_preempt12:20:131
1611699461,37cyclictest28886-21ssh11:10:132
1611699452,2cyclictest1192-21apt-get10:55:152
1611699451,41cyclictest6729-21ssh09:30:122
1611699451,2cyclictest0-21swapper/211:45:132
1610799453,6cyclictest131rcu_preempt07:25:110
16107994524,12cyclictest131rcu_preempt08:45:130
1610799451,35cyclictest1930-21ssh09:45:120
1611699442,38cyclictest0-21swapper/211:55:112
1611699442,2cyclictest1932-21apt-get09:45:132
1611699442,2cyclictest10372-21apt-get11:00:112
1611699441,3cyclictest0-21swapper/210:50:132
1611699441,39cyclictest0-21swapper/211:40:122
16107994415,22cyclictest12-21ksoftirqd/009:10:130
1611699432,37cyclictest0-21swapper/210:25:122
1611699431,37cyclictest0-21swapper/208:50:122
1611699421,2cyclictest0-21swapper/207:25:112
1611099422,8cyclictest131rcu_preempt12:25:121
16107994214,23cyclictest12-21ksoftirqd/007:40:130
1612499413,26cyclictest35966irq/58-eth011:10:133
1611699413,2cyclictest29272-21apt-get10:35:132
1611699411,3cyclictest0-21swapper/208:45:142
1612499404,23cyclictest35966irq/58-eth008:30:143
1612499402,23cyclictest35966irq/58-eth009:15:143
1612499402,23cyclictest35966irq/58-eth009:15:143
1611699401,2cyclictest22712-21apt-get12:35:122
1611699394,31cyclictest4426-21apt-get08:05:132
1611699391,3cyclictest0-21swapper/207:45:132
1611699391,34cyclictest0-21swapper/211:15:142
1611699391,34cyclictest0-21swapper/211:15:142
1611699391,2cyclictest14007-21apt-get08:15:112
1611699391,2cyclictest0-21swapper/207:30:152
1611699391,2cyclictest0-21swapper/207:30:152
16116993911,20cyclictest6143-21sh10:40:152
16116993911,20cyclictest6143-21sh10:40:152
1610799396,23cyclictest14826-21ssh11:20:120
16107993918,12cyclictest131rcu_preempt10:15:120
1611699381,4cyclictest0-21swapper/207:15:132
1611699381,3cyclictest0-21swapper/208:10:122
1611099382,26cyclictest22726-21ssh12:35:121
1610799387,8cyclictest131rcu_preempt07:30:150
1610799387,8cyclictest131rcu_preempt07:30:150
1610799382,27cyclictest12-21ksoftirqd/009:05:130
16124993724,7cyclictest35966irq/58-eth010:40:133
16124993724,7cyclictest35966irq/58-eth010:40:123
1611699371,3cyclictest29566-21apt-get10:00:142
1611699371,3cyclictest29566-21apt-get10:00:142
1611699371,3cyclictest23414-21apt-get12:00:152
1611699371,3cyclictest0-21swapper/208:35:132
1611699371,2cyclictest0-21swapper/210:10:152
1611699371,11cyclictest10048-21ntp_192.168.11410:05:262
1611099373,5cyclictest131rcu_preempt09:20:131
1610799371,31cyclictest15214-21ssh10:45:140
1610799371,25cyclictest20305-21rm09:55:110
1611699361,3cyclictest0-21swapper/209:25:142
1611699361,3cyclictest0-21swapper/207:50:122
1611699360,32cyclictest19722-21apt-get11:05:132
1611699360,32cyclictest15189-21ssh10:45:122
1611699360,2cyclictest0-21swapper/209:55:132
1611099362,28cyclictest5981-21ssh10:40:111
1611099362,28cyclictest5981-21ssh10:40:111
1610799362,26cyclictest19191-21ssh11:40:130
1611699354,28cyclictest861-21apt-get11:30:132
1611699351,6cyclictest31737-21apt-get08:00:012
1611699351,31cyclictest20087-21unixbench_singl09:35:302
1611699351,24cyclictest7350-21kthreadcore08:40:212
1611699351,24cyclictest7350-21kthreadcore08:40:212
1610799356,18cyclictest1204-21ssh10:55:130
1610799354,7cyclictest131rcu_preempt07:50:110
1612499346,14cyclictest35966irq/58-eth012:15:153
1612499341,13cyclictest0-21swapper/311:40:133
1611699341,6cyclictest11041-21/usr/sbin/munin09:50:292
1611699341,6cyclictest11041-21/usr/sbin/munin09:50:292
1611699341,4cyclictest29287-21unixbench_multi09:40:292
1611699341,2cyclictest0-21swapper/212:20:142
1611699340,8cyclictest10732-21kernelversion11:35:172
1611099342,28cyclictest5622-21ssh11:15:131
1611099342,28cyclictest5622-21ssh11:15:131
1611699331,7cyclictest15181-21if_eth111:20:182
1611699330,2cyclictest1701-21apt-get10:20:132
1611699326,24cyclictest31558-21ssh10:17:312
1611699321,7cyclictest19949-21ssh10:30:012
1611699321,3cyclictest0-21swapper/212:16:532
1611699321,23cyclictest28449-21/usr/sbin/munin08:30:302
1611699320,5cyclictest16446-21ssh09:15:402
1611699320,5cyclictest16446-21ssh09:15:402
1611699320,2cyclictest4465-21ssh12:25:142
1611099329,14cyclictest536-21usb-storage12:10:121
1611099329,14cyclictest536-21usb-storage12:10:121
1611099323,21cyclictest29587-21ssh10:00:141
1611099323,21cyclictest29587-21ssh10:00:141
16110993222,5cyclictest1767-21diskmemload12:05:131
1610799322,23cyclictest0-21swapper/010:40:130
1610799322,23cyclictest0-21swapper/010:40:130
1610799321,24cyclictest29295-21ssh10:35:140
1612499311,15cyclictest35966irq/58-eth010:25:133
1611699313,25cyclictest24953-21tune2fs09:00:162
1611699312,24cyclictest0-21swapper/207:10:132
1611699311,7cyclictest1725-21cut12:05:192
1611699311,6cyclictest29553-21basename09:25:012
1611699311,6cyclictest23903-21ssh11:25:042
16116993116,8cyclictest27-21ksoftirqd/209:10:122
1611699310,2cyclictest32059-21apt-get08:00:132
1612499302,16cyclictest35966irq/58-eth010:45:123
1611699291,7cyclictest28234-21sh08:30:012
1611699290,5cyclictest30689-21cut09:05:202
1611099291,20cyclictest5059-21sh11:50:141
1610799291,19cyclictest20137-21ssh10:30:140
16107992911,9cyclictest131rcu_preempt11:55:120
16110992816,8cyclictest16100-21cyclictest11:10:131
16110992814,7cyclictest21-21ksoftirqd/110:10:151
1611099281,18cyclictest10405-21ssh11:00:131
1610799282,20cyclictest5061-21ssh11:50:140
16107992812,11cyclictest0-21swapper/010:05:130
16124992711,13cyclictest0-21swapper/312:30:143
1611099272,20cyclictest23444-21ssh12:00:151
1611099272,19cyclictest24378-21ssh10:50:131
1611099272,19cyclictest1965-21ssh09:45:141
1611099272,17cyclictest24903-21ssh10:15:121
1611099271,20cyclictest19554-21ssh09:35:251
1611099270,18cyclictest1728-21ssh10:20:141
1610799273,19cyclictest23493-21ssh12:15:340
16107992614,4cyclictest12-21ksoftirqd/012:00:140
1610799261,18cyclictest25194-21ssh09:40:140
1612499251,18cyclictest0-21swapper/309:45:123
1610799256,15cyclictest12-21ksoftirqd/008:55:130
1610799252,15cyclictest6712-21ssh09:30:120
1612499241,14cyclictest0-21swapper/311:20:143
1611699241,5cyclictest16230-21latency_hist07:10:012
1611099241,17cyclictest25193-21sh09:40:141
1611099241,16cyclictest14839-21sh11:20:141
16107992413,8cyclictest20817-21cron07:15:010
16107992412,9cyclictest24752-21cron09:40:010
1612499235,14cyclictest35966irq/58-eth010:30:143
1612499232,17cyclictest35966irq/58-eth011:30:133
1612499232,13cyclictest0-21swapper/311:15:133
1612499232,13cyclictest0-21swapper/311:15:133
1610799233,13cyclictest11076-21ssh09:50:130
1610799233,13cyclictest11076-21ssh09:50:130
1610799231,17cyclictest20725-21sh09:20:140
1612499222,10cyclictest0-21swapper/307:15:143
1611099223,13cyclictest0-21swapper/108:30:141
1611099221,17cyclictest16296-21ls07:40:281
1611099221,16cyclictest27282-21ls08:25:281
1610799221,18cyclictest1229-21ssh09:25:270
1612499215,10cyclictest35966irq/58-eth012:10:113
1612499215,10cyclictest35966irq/58-eth012:10:113
1612499211,15cyclictest0-21swapper/309:20:143
1611099217,4cyclictest131rcu_preempt09:55:211
1611099214,10cyclictest0-21swapper/107:20:131
16110992115,4cyclictest1049-21sh10:55:051
1611099211,17cyclictest25058-21ls09:00:171
1611099211,15cyclictest15222-21apt-get08:50:131
16110992110,9cyclictest15527-21cron09:35:011
1611099210,18cyclictest1767-21diskmemload12:17:471
1610799218,8cyclictest131rcu_preempt08:20:140
1610799213,10cyclictest0-21swapper/008:50:130
16107992115,4cyclictest0-21swapper/012:35:120
1610799211,15cyclictest8163-21ssh12:25:270
1610799211,14cyclictest9351-21sh12:10:130
1610799211,14cyclictest9351-21sh12:10:130
1610799211,14cyclictest15797-21cut10:10:170
1612499209,8cyclictest0-21swapper/309:41:123
1612499201,15cyclictest35966irq/58-eth010:00:143
1612499201,15cyclictest35966irq/58-eth010:00:143
1611099206,4cyclictest15783-21tr11:55:201
1611099206,10cyclictest30199-21ssh10:35:201
1611099203,15cyclictest131rcu_preempt11:45:121
1611099203,11cyclictest24002-21kworker/u8:111:30:131
16110992011,7cyclictest17496-21sshd10:27:291
1611099201,16cyclictest0-21swapper/111:25:141
1611099201,15cyclictest15204-21ssh10:45:131
16110992011,3cyclictest1767-21diskmemload12:33:001
1611099200,15cyclictest9396-21timerwakeupswit08:40:291
1611099200,15cyclictest9396-21timerwakeupswit08:40:291
1610799203,12cyclictest1767-21diskmemload11:35:100
1610799202,2cyclictest131rcu_preempt11:25:320
1610799201,3cyclictest131rcu_preempt08:05:160
1610799201,3cyclictest131rcu_preempt07:20:120
1610799201,15cyclictest5425-21ssh10:20:280
1610799201,14cyclictest9397-21sh11:15:270
1610799201,14cyclictest9397-21sh11:15:270
1612499199,7cyclictest35966irq/58-eth009:55:133
1612499199,3cyclictest0-21swapper/310:07:203
16124991911,5cyclictest35966irq/58-eth011:25:293
1612499191,11cyclictest0-21swapper/308:55:163
16124991910,6cyclictest35966irq/58-eth010:20:143
16124991910,6cyclictest0-21swapper/311:37:123
16124991910,6cyclictest0-21swapper/311:03:213
1611099193,12cyclictest31728-21munin-run08:00:011
1611099193,12cyclictest15070-21ssh09:15:251
1611099193,12cyclictest15070-21ssh09:15:251
1611099193,11cyclictest29957-21ssh09:25:131
16110991912,4cyclictest28874-21idleruntime-cro10:35:011
1611099191,15cyclictest19179-21kthreadcore07:45:211
16110991911,5cyclictest14881-21processes09:50:281
16110991911,5cyclictest14881-21processes09:50:281
1610799198,8cyclictest31636-21awk12:20:300
1610799193,13cyclictest12-21ksoftirqd/010:50:120
1610799192,13cyclictest32666-21diskstats12:05:150
1610799191,9cyclictest131rcu_preempt08:00:230
1610799191,3cyclictest131rcu_preempt09:00:250
1610799191,13cyclictest9557-21sed08:10:170
1612499189,6cyclictest0-21swapper/310:53:483
1612499189,6cyclictest0-21swapper/310:35:333
1612499188,8cyclictest0-21swapper/311:45:393
1612499188,6cyclictest35966irq/58-eth012:25:133
1612499185,7cyclictest0-21swapper/307:20:133
1612499182,10cyclictest0-21swapper/307:50:043
1612499182,10cyclictest0-21swapper/307:35:143
1612499181,13cyclictest0-21swapper/309:05:013
1612499181,11cyclictest0-21swapper/311:55:303
1612499181,11cyclictest0-21swapper/310:55:053
1612499181,10cyclictest0-21swapper/308:50:143
16124991810,6cyclictest0-21swapper/310:10:303
16124991810,6cyclictest0-21swapper/309:52:073
16124991810,6cyclictest0-21swapper/309:52:073
16124991810,6cyclictest0-21swapper/309:30:303
1611099186,9cyclictest14540-21irqrtprio08:15:161
1611099183,12cyclictest24862-21sed07:15:291
1611099183,11cyclictest0-21swapper/108:20:151
1611099182,13cyclictest13719-21sed11:35:241
1611099182,11cyclictest0-21swapper/107:10:141
1611099181,8cyclictest32041-21/usr/sbin/munin08:00:141
1611099181,14cyclictest7834-21/usr/sbin/munin07:35:281
1611099181,14cyclictest3070-21apt-get07:30:151
1611099181,14cyclictest3070-21apt-get07:30:141
1611099181,10cyclictest10610-21fschecks_count08:45:161
1610799188,7cyclictest32099-21cut11:10:230
1610799183,10cyclictest151rcuc/010:00:130
1610799183,10cyclictest151rcuc/010:00:120
1610799182,11cyclictest0-21swapper/008:25:140
1610799181,6cyclictest11288-21irqcore10:25:170
1610799181,14cyclictest19240-21apt-get11:05:010
16107991810,3cyclictest12-21ksoftirqd/011:05:030
1612499179,6cyclictest0-21swapper/309:39:423
1612499179,6cyclictest0-21swapper/309:12:283
1612499178,6cyclictest0-21swapper/311:05:053
1612499172,10cyclictest0-21swapper/309:05:143
16124991711,4cyclictest0-21swapper/312:35:143
1612499171,12cyclictest0-21swapper/307:45:233
1612499171,10cyclictest0-21swapper/307:40:143
1612499171,10cyclictest0-21swapper/307:30:163
1612499171,10cyclictest0-21swapper/307:30:163
16124991710,4cyclictest35966irq/58-eth012:00:153
16124991710,3cyclictest0-21swapper/311:50:133
1612499170,10cyclictest0-21swapper/308:40:153
1612499170,10cyclictest0-21swapper/308:40:153
1611099174,10cyclictest7935-21proc_pri08:05:271
1611099173,10cyclictest0-21swapper/109:05:131
1611099172,12cyclictest8854-21sh10:05:221
1611099172,10cyclictest0-21swapper/111:05:141
1611099171,13cyclictest19555-21irqcore11:40:171
16110991710,4cyclictest2902-21kernelversion09:10:191
1610799177,3cyclictest131rcu_preempt07:15:180
1610799172,12cyclictest840-21apt-get08:35:110
1610799172,11cyclictest0-21swapper/007:55:130
1610799171,5cyclictest20858-21ntp_192.168.11407:45:240
16107991711,4cyclictest11423-21sendmail_mailqu07:35:280
1612499167,6cyclictest0-21swapper/308:15:113
1612499163,10cyclictest36066irq/59-eth010:15:123
1612499162,9cyclictest0-21swapper/308:20:153
1612499161,12cyclictest0-21swapper/307:55:163
1612499161,10cyclictest0-21swapper/308:25:133
1611099161,13cyclictest840-21apt-get08:35:121
1611099161,12cyclictest24721-21kthreadcore07:50:191
1611099161,11cyclictest9360-21fschecks_count08:10:151
1611099161,11cyclictest31322-21uname07:25:171
16110991610,4cyclictest0-21swapper/108:55:171
1610799168,5cyclictest31973-21proc_pri08:30:280
1612499152,7cyclictest35966irq/58-eth008:05:123
1612499151,6cyclictest35966irq/58-eth007:25:113
1612499151,11cyclictest0-21swapper/308:35:123
1612499151,10cyclictest0-21swapper/308:10:133
1611099156,6cyclictest91-21pr/ttyS007:10:011
1610799151,4cyclictest131rcu_preempt08:15:100
1610799151,11cyclictest16169-21cat07:10:000
1612499141,9cyclictest35966irq/58-eth007:10:143
1612499130,8cyclictest0-21swapper/308:45:143
1612499122,8cyclictest0-21swapper/308:05:013
1612499102,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional