You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 13:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Aug 26, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1780899713,51cyclictest5554-21ssh22:05:150
17833996643,16cyclictest35966irq/58-eth023:55:133
17808996614,9cyclictest131rcu_preempt19:30:150
17808996210,7cyclictest13050-21ssh21:15:140
17808995915,8cyclictest131rcu_preempt23:50:140
1780899503,7cyclictest131rcu_preempt19:45:130
1780899503,37cyclictest20550-21ssh00:20:130
1780899492,34cyclictest16503-21ssh21:35:130
1780899492,34cyclictest16503-21ssh21:35:130
1780899480,8cyclictest131rcu_preempt00:30:140
17833994723,17cyclictest35966irq/58-eth021:10:133
17825994728,13cyclictest6139-21apt-get00:30:142
1781699471,41cyclictest15744-21ssh23:05:111
17825994627,15cyclictest31010-21apt-get21:25:132
1782599461,32cyclictest26782-21ssh22:35:122
1783399452,30cyclictest35966irq/58-eth022:40:123
1783399443,29cyclictest35966irq/58-eth021:20:133
17825994432,8cyclictest28194-21apt-get23:30:152
1780899443,31cyclictest22048-21ssh21:20:140
1782599432,30cyclictest930-21snmpd22:50:142
17808994324,9cyclictest131rcu_preempt20:55:130
1782599427,17cyclictest0-21swapper/223:00:142
1783399414,24cyclictest35966irq/58-eth023:50:133
1783399412,27cyclictest35966irq/58-eth022:00:143
1781699412,30cyclictest3287-21ssh22:40:121
1780899413,22cyclictest4034-21ssh21:10:150
1781699403,30cyclictest2033-21ssh21:45:121
1780899401,28cyclictest25516-21ssh21:40:130
1780899401,28cyclictest25516-21ssh21:40:130
1782599399,24cyclictest0-21swapper/220:20:122
1782599398,15cyclictest6543-21cat00:10:282
17825993922,12cyclictest14492-21ssh22:10:142
1783399382,24cyclictest35966irq/58-eth021:50:133
1782599389,19cyclictest19639-21ssh22:48:282
1782599387,13cyclictest13515-21cut21:15:192
17825993822,11cyclictest31457-21ssh23:50:132
17825993811,20cyclictest0-21swapper/221:50:142
17825993810,12cyclictest9157-21sort22:25:182
1780899383,32cyclictest13647-21ssh23:40:130
1783399375,21cyclictest35966irq/58-eth021:35:133
1783399375,21cyclictest35966irq/58-eth021:35:133
1782599379,18cyclictest22133-21if_eth020:55:212
1782599377,12cyclictest930-21snmpd23:15:192
1782599377,12cyclictest930-21snmpd23:15:182
17825993723,10cyclictest30355-21apt-get22:55:132
17825993722,9cyclictest16943-21apt-get00:00:142
1781699371,30cyclictest1328-21ssh23:15:121
1781699371,30cyclictest1328-21ssh23:15:121
1783399363,17cyclictest35966irq/58-eth000:00:133
1782599367,15cyclictest13206-21cat00:15:222
17825993617,15cyclictest20011-21apt-get21:55:132
17825993610,14cyclictest28277-21grep21:00:192
1781699362,21cyclictest21-21ksoftirqd/119:25:131
1780899364,21cyclictest12-21ksoftirqd/023:55:130
1780899361,28cyclictest30387-21rm22:55:140
1783399352,22cyclictest35966irq/58-eth000:25:133
1782599356,11cyclictest0-21swapper/219:15:242
17825993525,6cyclictest25971-21ssh00:05:142
1780899352,25cyclictest17880-21ssh22:30:130
17808993518,11cyclictest131rcu_preempt20:35:140
1782599349,17cyclictest15411-21usage_EthcTask20:10:312
1782599346,13cyclictest7238-21latency_hist20:40:012
1781699341,4cyclictest0-21swapper/123:10:111
1780899345,22cyclictest12-21ksoftirqd/023:00:130
1780899341,25cyclictest11073-21sh21:50:130
1782599337,9cyclictest16422-21ssh21:35:092
1782599337,9cyclictest16422-21ssh21:35:092
1782599337,13cyclictest31334-21cat19:20:262
1782599336,11cyclictest30236-21ssh22:17:562
1782599335,18cyclictest749-21ssh22:00:312
1782599335,18cyclictest3621-21diskmemload22:06:372
1782599334,17cyclictest0-21swapper/223:47:572
17816993313,8cyclictest0-21swapper/119:15:141
1782599328,14cyclictest12008-21rm21:14:152
1782599326,9cyclictest20246-21kthreadcore19:10:202
1782599326,18cyclictest0-21swapper/221:45:122
1782599326,12cyclictest30534-21irqcore20:30:182
1782599326,10cyclictest23572-21cat23:25:302
1782599325,11cyclictest3621-21diskmemload23:58:372
1782599324,18cyclictest25502-21ssh21:40:142
1782599324,18cyclictest25502-21ssh21:40:132
1782599322,14cyclictest13259-21latency_hist23:40:012
17808993215,11cyclictest12-21ksoftirqd/023:25:120
1782599316,17cyclictest3621-21diskmemload22:41:002
1782599316,11cyclictest31126-21ssh23:12:212
1782599315,13cyclictest31548-21df21:05:232
1782599315,10cyclictest0-21swapper/223:06:182
17825993114,13cyclictest0-21swapper/220:25:122
1782599311,15cyclictest0-21swapper/221:30:152
1780899311,23cyclictest29577-21ssh00:25:150
1782599308,10cyclictest5767-21ssh22:21:412
1782599306,13cyclictest22385-21if_eth021:20:182
1782599305,16cyclictest0-21swapper/222:34:342
1782599304,7cyclictest0-21swapper/200:26:422
1780899309,18cyclictest12-21ksoftirqd/021:00:130
1780899301,21cyclictest11536-21ssh00:15:150
1782599298,11cyclictest5993-21sed20:05:022
1782599296,14cyclictest15670-21/usr/sbin/munin20:15:252
1782599294,7cyclictest3621-21diskmemload23:20:032
1782599293,12cyclictest3621-21diskmemload23:43:062
17825992913,8cyclictest9755-21grep20:05:292
1780899295,12cyclictest29042-21ssh22:00:140
1783399289,12cyclictest35966irq/58-eth023:20:133
17833992812,9cyclictest35966irq/58-eth022:55:153
1782599286,7cyclictest29381-21irqrtprio19:55:182
1782599285,9cyclictest21111-21ps00:20:192
1782599285,8cyclictest9642-21apt-get19:35:122
17825992812,13cyclictest4580-21sshd19:25:592
1781699283,10cyclictest21-21ksoftirqd/123:00:301
1781699280,6cyclictest13040-21apt-get21:15:141
1783399273,17cyclictest35966irq/58-eth023:40:143
1783399273,15cyclictest0-21swapper/321:18:003
1783399272,13cyclictest35966irq/58-eth022:10:133
1783399271,20cyclictest0-21swapper/321:45:133
1782599278,15cyclictest0-21swapper/219:30:142
1782599277,11cyclictest0-21swapper/220:45:282
1782599273,17cyclictest0-21swapper/200:35:152
1781699277,4cyclictest131rcu_preempt00:10:141
1781699276,5cyclictest21-21ksoftirqd/123:55:271
1781699274,5cyclictest21-21ksoftirqd/123:40:321
1781699274,5cyclictest21-21ksoftirqd/122:35:281
1781699272,4cyclictest21-21ksoftirqd/123:25:181
17816992717,5cyclictest29554-21apt-get00:25:131
1780899272,19cyclictest22807-21ssh00:01:410
1783399261,16cyclictest35966irq/58-eth000:15:143
1782599268,7cyclictest18456-21cat19:40:282
1781699267,9cyclictest21-21ksoftirqd/121:59:591
1781699267,5cyclictest21-21ksoftirqd/100:18:431
1781699266,13cyclictest16945-21ssh00:00:131
1781699265,8cyclictest21-21ksoftirqd/122:00:221
1781699265,6cyclictest21-21ksoftirqd/122:15:281
1781699265,4cyclictest21-21ksoftirqd/121:25:231
1781699264,9cyclictest131rcu_preempt21:24:341
1781699264,6cyclictest21-21ksoftirqd/122:55:001
1781699264,5cyclictest21-21ksoftirqd/123:20:141
1781699263,5cyclictest21-21ksoftirqd/120:45:191
1781699263,10cyclictest21-21ksoftirqd/122:25:241
1781699262,6cyclictest21-21ksoftirqd/122:49:201
1781699262,18cyclictest14502-21ssh22:10:161
1781699261,7cyclictest21-21ksoftirqd/100:30:271
1781699261,5cyclictest21-21ksoftirqd/120:05:281
1781699261,17cyclictest15169-21ssh00:35:141
17833992513,9cyclictest0-21swapper/322:05:143
1782599254,7cyclictest0-21swapper/220:50:292
1781699257,9cyclictest21-21ksoftirqd/123:47:451
1781699257,6cyclictest21-21ksoftirqd/123:35:021
1781699253,6cyclictest21-21ksoftirqd/122:30:141
1781699253,6cyclictest21-21ksoftirqd/122:21:371
17816992520,4cyclictest201rcuc/123:50:081
17816992519,4cyclictest201rcuc/122:06:241
17816992517,4cyclictest11060-21apt-get21:50:141
17816992514,8cyclictest4015-21apt-get21:10:141
1780899256,12cyclictest12-21ksoftirqd/019:20:130
1780899251,19cyclictest10311-21ssh23:20:120
17833992413,7cyclictest35966irq/58-eth023:05:133
1782599246,7cyclictest27488-21ntp_192.168.11419:50:232
1781699244,6cyclictest21-21ksoftirqd/123:35:481
1781699244,5cyclictest21-21ksoftirqd/121:36:411
1781699244,5cyclictest21-21ksoftirqd/121:36:411
1781699244,5cyclictest21-21ksoftirqd/119:30:281
1781699244,5cyclictest21-21ksoftirqd/100:20:271
1781699244,3cyclictest21-21ksoftirqd/119:10:021
1781699243,6cyclictest21-21ksoftirqd/100:07:131
1781699243,4cyclictest21-21ksoftirqd/119:35:211
1781699241,6cyclictest21-21ksoftirqd/121:31:321
1781699241,14cyclictest0-21swapper/120:30:151
1780899249,7cyclictest0-21swapper/021:25:130
1780899242,18cyclictest20044-21ssh21:55:140
1781699235,3cyclictest131rcu_preempt22:55:051
1781699234,5cyclictest21-21ksoftirqd/119:40:301
1781699234,10cyclictest131rcu_preempt20:40:231
1781699233,5cyclictest21-21ksoftirqd/121:42:121
1781699233,5cyclictest21-21ksoftirqd/121:42:111
1781699233,5cyclictest21-21ksoftirqd/120:50:281
1781699233,5cyclictest21-21ksoftirqd/119:20:191
1781699233,5cyclictest21-21ksoftirqd/119:10:161
1781699232,5cyclictest21-21ksoftirqd/120:20:011
1781699231,5cyclictest21-21ksoftirqd/119:45:271
1780899232,2cyclictest131rcu_preempt19:25:120
17808992314,7cyclictest0-21swapper/023:10:220
17808992313,3cyclictest0-21swapper/022:25:130
1780899231,18cyclictest0-21swapper/020:10:140
17833992210,9cyclictest0-21swapper/322:29:373
1782599226,13cyclictest19959-21ps19:45:202
17825992210,9cyclictest135-21systemd-journal20:40:012
1781699226,7cyclictest25276-21ntpq20:55:281
1781699226,4cyclictest21-21ksoftirqd/121:00:141
1781699225,3cyclictest131rcu_preempt21:05:371
1781699225,3cyclictest131rcu_preempt20:20:021
1781699224,9cyclictest21-21ksoftirqd/119:55:191
1781699224,4cyclictest21-21ksoftirqd/119:50:211
1781699224,10cyclictest21-21ksoftirqd/120:00:151
1781699222,4cyclictest21-21ksoftirqd/120:35:251
1781699221,4cyclictest21-21ksoftirqd/120:25:351
17816992214,4cyclictest10879-21apt-get20:10:131
1780899227,4cyclictest131rcu_preempt19:55:220
1780899221,16cyclictest23422-21ssh22:15:150
17833992111,6cyclictest35966irq/58-eth021:40:143
17833992111,6cyclictest35966irq/58-eth021:40:133
1780899217,10cyclictest131rcu_preempt00:05:140
1780899216,3cyclictest131rcu_preempt21:35:020
1780899211,4cyclictest131rcu_preempt23:06:100
1780899211,4cyclictest131rcu_preempt19:35:120
1780899211,2cyclictest131rcu_preempt00:36:500
17808992111,7cyclictest4927-21cut21:45:220
1783399203,12cyclictest0-21swapper/321:30:003
1783399203,11cyclictest0-21swapper/300:10:023
1783399202,11cyclictest35966irq/58-eth022:35:123
1783399201,15cyclictest0-21swapper/300:35:143
1780899209,4cyclictest12-21ksoftirqd/020:00:160
1780899203,4cyclictest131rcu_preempt22:40:140
1780899203,13cyclictest12-21ksoftirqd/019:40:140
1780899201,3cyclictest131rcu_preempt23:49:490
1780899201,15cyclictest21685-21ls20:55:010
1780899200,15cyclictest12378-21grep22:45:150
1780899200,10cyclictest3473-21kthreadcore23:15:210
1780899200,10cyclictest3473-21kthreadcore23:15:210
1783399199,7cyclictest35966irq/58-eth000:05:153
1783399198,9cyclictest0-21swapper/323:28:583
1783399195,8cyclictest0-21swapper/320:35:143
1783399192,11cyclictest0-21swapper/320:55:143
1783399191,13cyclictest0-21swapper/319:45:173
1783399191,13cyclictest0-21swapper/319:15:163
1783399191,10cyclictest35966irq/58-eth022:20:103
17833991910,6cyclictest0-21swapper/323:18:053
17833991910,6cyclictest0-21swapper/323:18:053
17833991910,6cyclictest0-21swapper/321:55:143
17833991910,5cyclictest35966irq/58-eth022:30:133
1782599198,8cyclictest17884-21cat19:10:022
1780899199,2cyclictest12-21ksoftirqd/020:30:020
1780899198,8cyclictest19561-21grep20:15:280
1780899196,3cyclictest131rcu_preempt23:36:190
1780899194,3cyclictest131rcu_preempt20:45:200
1780899193,3cyclictest131rcu_preempt20:40:210
1780899192,3cyclictest131rcu_preempt23:30:160
1780899191,3cyclictest192371kthreadcore19:10:200
1780899191,3cyclictest131rcu_preempt00:12:490
17808991912,5cyclictest29976-21cron22:55:020
1780899191,14cyclictest3535-21ssh22:20:230
1780899190,3cyclictest131rcu_preempt19:15:140
1783399184,10cyclictest0-21swapper/300:30:153
1783399183,10cyclictest0-21swapper/300:21:033
1783399182,11cyclictest0-21swapper/321:35:003
1783399182,10cyclictest0-21swapper/320:40:143
1783399182,10cyclictest0-21swapper/320:25:143
1783399182,10cyclictest0-21swapper/320:00:143
1783399182,10cyclictest0-21swapper/319:50:153
17833991812,4cyclictest0-21swapper/323:30:233
1783399181,13cyclictest35966irq/58-eth023:45:133
17833991810,6cyclictest0-21swapper/323:03:433
17833991810,6cyclictest0-21swapper/322:18:273
1780899182,9cyclictest12-21ksoftirqd/022:10:130
1780899181,3cyclictest131rcu_preempt20:30:300
1780899181,3cyclictest131rcu_preempt19:50:120
17808991813,3cyclictest0-21swapper/022:36:120
1780899181,2cyclictest131rcu_preempt20:20:050
1780899180,3cyclictest131rcu_preempt20:05:300
1783399179,6cyclictest0-21swapper/322:54:133
1783399179,3cyclictest0-21swapper/322:45:143
1783399176,8cyclictest0-21swapper/323:35:313
1783399174,9cyclictest35966irq/58-eth020:20:123
1783399172,10cyclictest0-21swapper/320:30:143
1783399172,10cyclictest0-21swapper/320:05:153
1783399172,10cyclictest0-21swapper/319:40:143
1783399172,10cyclictest0-21swapper/319:35:143
1783399171,13cyclictest0-21swapper/319:25:273
1783399171,10cyclictest0-21swapper/320:45:143
17833991710,4cyclictest0-21swapper/323:10:123
1780899178,7cyclictest31464-21apt-get21:05:140
1783399157,6cyclictest0-21swapper/319:10:263
1783399152,10cyclictest0-21swapper/321:05:153
1783399141,8cyclictest0-21swapper/320:50:333
1783399141,11cyclictest0-21swapper/320:20:033
1780899141,11cyclictest17843-21latency_hist19:10:020
1783399137,4cyclictest0-21swapper/321:00:243
1783399132,8cyclictest0-21swapper/320:15:013
1783399131,7cyclictest0-21swapper/319:30:183
1783399123,5cyclictest0-21swapper/319:20:123
1783399121,9cyclictest0-21swapper/319:55:293
1783399111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional