You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 05:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Jun 09, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1489799701,60cyclictest0-21swapper/220:15:122
1487899632,41cyclictest17287-21ssh22:50:110
14878995912,9cyclictest131rcu_preempt20:20:110
1487899573,5cyclictest131rcu_preempt19:55:140
1488799551,44cyclictest7354-21ssh21:50:151
1489799532,46cyclictest0-21swapper/223:40:122
1489799532,46cyclictest0-21swapper/223:40:122
1489799521,45cyclictest0-21swapper/200:35:122
1487899515,33cyclictest4001-21ssh21:30:130
1487899514,34cyclictest131rcu_preempt23:30:120
1489799502,35cyclictest26147-21apt-get22:55:132
14878995011,6cyclictest131rcu_preempt20:50:130
1487899491,39cyclictest26175-21ssh22:55:130
1489799481,43cyclictest6919-21apt-get00:15:142
1489799481,40cyclictest0-21swapper/223:30:122
1489799471,32cyclictest13391-21grep21:35:182
1489799471,30cyclictest8819-21grep22:25:272
1489799471,30cyclictest8819-21grep22:25:262
1487899474,28cyclictest20558-21ssh23:10:120
1489799460,30cyclictest9216-21ssh22:08:452
1487899464,32cyclictest22964-21ssh22:35:130
1489799451,29cyclictest28460-21/usr/sbin/munin21:05:162
1489799451,28cyclictest26962-21ssh23:50:132
1489799445,25cyclictest5058-21kthreadcore23:00:222
1489799445,25cyclictest5058-21kthreadcore23:00:222
1489799440,39cyclictest0-21swapper/221:10:142
1489799440,28cyclictest6643-21grep21:30:222
1489799431,37cyclictest22233-21apt-get20:25:122
1489799431,33cyclictest25207-21ssh22:00:132
1489799431,29cyclictest30091-21kthreadcore19:25:202
1489799430,2cyclictest21989-21apt-get21:40:142
1487899434,7cyclictest131rcu_preempt23:45:120
1489799423,34cyclictest0-21swapper/223:55:122
1489799421,36cyclictest11703-21apt-get23:05:152
1489799420,3cyclictest21430-21apt-get00:05:142
1489799420,2cyclictest14715-21ssh23:25:112
1489799410,28cyclictest4081-21ssh22:24:152
1489799410,27cyclictest26621-21tr19:20:202
1489799410,26cyclictest27776-21cut21:25:152
1488799415,26cyclictest28573-21ssh22:20:141
1487899416,30cyclictest11725-21ssh23:05:150
1490599402,22cyclictest35966irq/58-eth021:25:133
1489799402,34cyclictest0-21swapper/219:55:132
1489799402,33cyclictest0-21swapper/223:20:132
1489799402,27cyclictest16409-21fschecks_count19:45:172
1488799401,36cyclictest0-21swapper/123:35:121
1487899404,23cyclictest30809-21ssh21:45:140
1490599394,21cyclictest35966irq/58-eth022:25:123
1490599394,21cyclictest35966irq/58-eth022:25:123
1489799391,34cyclictest0-21swapper/219:30:052
1489799391,33cyclictest0-21swapper/221:20:112
1489799391,24cyclictest3233-21ntp_states22:40:262
1489799391,24cyclictest22323-21ssh21:56:542
1489799382,30cyclictest566-21cut23:35:162
1489799381,4cyclictest30368-21apt-get00:10:122
1489799381,34cyclictest17450-21switchtime20:50:282
1489799381,30cyclictest17266-21apt-get22:50:112
1489799381,27cyclictest0-21swapper/222:15:112
1489799381,20cyclictest1604-21ntp_192.168.11420:00:232
1490599372,21cyclictest35966irq/58-eth023:45:133
1490599370,22cyclictest35966irq/58-eth020:00:123
1489799372,30cyclictest27346irq/40-dwc2_hso20:40:132
1489799371,33cyclictest3970-21sendmail20:40:012
1489799370,5cyclictest20841-21sh22:32:542
1489799370,3cyclictest20547-21ssh23:10:122
1489799370,2cyclictest1377-21apt-get00:30:132
1488799373,29cyclictest18640-21ssh21:20:131
14887993715,19cyclictest0-21swapper/123:55:121
1487899372,22cyclictest131rcu_preempt23:50:120
1487899371,25cyclictest10635-21ssh22:10:120
1490599363,19cyclictest35966irq/58-eth021:35:123
1489799361,32cyclictest13473-21kthreadcore00:00:202
1489799361,31cyclictest665466/usr/sbin/munin19:35:122
1489799361,31cyclictest18868-21sed19:10:292
1489799361,30cyclictest25019-21fschecks_time00:25:162
1489799361,30cyclictest25019-21fschecks_time00:25:162
1489799360,2cyclictest0-21swapper/223:45:292
1489799351,31cyclictest7785-21/usr/sbin/munin20:10:242
1489799351,31cyclictest7785-21/usr/sbin/munin20:10:242
1489799351,31cyclictest24035-21cut21:00:182
1489799351,31cyclictest18237-21cron21:20:022
1489799351,31cyclictest12108-21ntp_kernel_err22:45:252
1489799351,26cyclictest12698-21/usr/sbin/munin20:45:292
1489799350,31cyclictest21536-21ssh00:21:302
1489799350,2cyclictest0-21swapper/220:05:152
14878993523,6cyclictest121ksoftirqd/023:25:130
1489799342,20cyclictest17403-21/usr/sbin/munin20:20:212
1489799341,22cyclictest0-21swapper/221:45:112
14878993417,10cyclictest131rcu_preempt22:15:110
14878993415,14cyclictest12-21ksoftirqd/022:00:120
1489799331,23cyclictest22926-21/usr/sbin/munin22:35:222
1489799330,30cyclictest21880-21sed20:55:212
1489799330,2cyclictest0-21swapper/220:30:122
1487899332,23cyclictest567-21sed23:35:170
1487899331,24cyclictest339-21diskmemload23:15:100
1489799321,28cyclictest21044-21kthreadcore19:15:182
1489799321,27cyclictest29767-21/usr/sbin/munin23:15:152
1489799320,30cyclictest0-21swapper/222:15:002
1489799320,29cyclictest25604-21sshd19:54:162
1488799321,24cyclictest1282-21kworker/u8:022:30:121
1487899321,19cyclictest1742-21kworker/u8:100:00:120
14905993126,3cyclictest35966irq/58-eth023:35:123
1489799310,28cyclictest16065-21awk19:45:002
1489799310,18cyclictest0-21swapper/221:50:082
1488799312,25cyclictest9056-21apt-get23:40:121
1488799312,25cyclictest9056-21apt-get23:40:121
1487899310,23cyclictest0-21swapper/021:00:120
1488799302,20cyclictest2784-21ssh23:00:131
1488799302,20cyclictest2784-21ssh23:00:121
1487899302,22cyclictest9674-21ssh21:15:130
1487899301,18cyclictest7345-21ssh21:50:150
1488799291,6cyclictest0-21swapper/123:05:141
1488799291,22cyclictest6946-21ssh00:15:151
1490599282,19cyclictest35966irq/58-eth000:10:123
1488799282,18cyclictest1408-21ssh00:30:121
1488799281,16cyclictest30380-21ssh00:10:111
1490599272,12cyclictest35966irq/58-eth020:35:143
1490599271,15cyclictest0-21swapper/323:30:113
1489799273,21cyclictest0-21swapper/219:10:012
1488799274,15cyclictest630-21Async09:36:311
1487899277,7cyclictest131rcu_preempt19:10:210
1487899273,14cyclictest21452-21ssh00:05:140
1490599261,12cyclictest35966irq/58-eth023:15:113
1488799262,17cyclictest15851-21diskmemload00:20:111
1488799261,19cyclictest8506-21ssh22:45:141
1487899268,10cyclictest131rcu_preempt20:10:140
1487899268,10cyclictest131rcu_preempt20:10:130
1487899264,13cyclictest12-21ksoftirqd/000:10:120
1487899262,20cyclictest9084-21ssh23:40:130
1487899262,20cyclictest9084-21ssh23:40:130
1488799259,5cyclictest131rcu_preempt21:10:181
1488799257,12cyclictest0-21swapper/121:00:181
1487899251,16cyclictest31984-21ssh22:40:140
1490599244,8cyclictest36066irq/59-eth000:00:123
1490599241,14cyclictest35966irq/58-eth021:50:143
1488799247,3cyclictest131rcu_preempt20:10:131
1488799247,3cyclictest131rcu_preempt20:10:131
1488799246,7cyclictest21-21ksoftirqd/123:53:501
1488799241,17cyclictest0-21swapper/122:35:131
14887992410,5cyclictest0-21swapper/122:55:501
1487899245,3cyclictest131rcu_preempt21:25:120
1490599231,12cyclictest35966irq/58-eth023:25:133
1488799238,7cyclictest21-21ksoftirqd/121:28:031
1488799237,4cyclictest21-21ksoftirqd/123:25:151
1488799237,4cyclictest0-21swapper/121:16:101
1488799235,3cyclictest131rcu_preempt22:13:441
1488799234,3cyclictest131rcu_preempt00:00:131
1488799234,11cyclictest12984-21ssh21:35:121
1488799231,17cyclictest1731-21ssh22:05:111
14887992311,3cyclictest0-21swapper/123:21:521
14887992311,10cyclictest0-21swapper/121:35:011
1487899236,4cyclictest131rcu_preempt20:30:270
1487899233,14cyclictest10397-21ssh00:35:140
14878992313,5cyclictest121ksoftirqd/021:35:130
1487899231,18cyclictest4196-21find20:40:130
1490599226,13cyclictest35966irq/58-eth022:05:113
1490599225,14cyclictest36066irq/59-eth022:30:123
1490599221,11cyclictest35966irq/58-eth021:15:113
1488799228,4cyclictest21-21ksoftirqd/123:30:281
1488799227,5cyclictest21-21ksoftirqd/123:46:191
1488799227,5cyclictest21-21ksoftirqd/120:40:001
1488799227,11cyclictest21-21ksoftirqd/119:25:131
1488799223,12cyclictest131rcu_preempt00:25:221
1488799223,12cyclictest131rcu_preempt00:25:221
14887992213,6cyclictest8894-21sendmail_mailtr22:25:271
14887992213,6cyclictest8894-21sendmail_mailtr22:25:261
1488799221,17cyclictest13793-21apt-get20:50:121
1488799221,16cyclictest31962-21apt-get22:40:131
1487899229,3cyclictest131rcu_preempt19:25:250
1487899223,8cyclictest0-21swapper/021:10:320
14878992212,7cyclictest12161-21ssh22:45:260
1487899221,15cyclictest5910-21find23:20:120
1487899220,18cyclictest18804-21fschecks_count21:20:150
14905992113,5cyclictest0-21swapper/323:00:133
14905992113,5cyclictest0-21swapper/323:00:133
14905992111,7cyclictest0-21swapper/323:10:333
14905992111,6cyclictest35966irq/58-eth021:55:123
14905992110,7cyclictest35966irq/58-eth022:50:123
1488799218,4cyclictest131rcu_preempt19:35:201
1488799218,3cyclictest131rcu_preempt22:50:101
1488799217,4cyclictest21-21ksoftirqd/121:46:151
1488799215,5cyclictest21-21ksoftirqd/122:02:331
1488799214,11cyclictest131rcu_preempt20:20:251
1488799213,3cyclictest131rcu_preempt23:10:141
1488799213,3cyclictest131rcu_preempt00:05:191
1488799212,3cyclictest131rcu_preempt19:45:011
14887992114,4cyclictest0-21swapper/119:10:151
1488799211,3cyclictest0-21swapper/121:40:211
1488799211,15cyclictest29513-21ssh23:15:131
14887992110,6cyclictest201rcuc/119:55:131
1488799210,4cyclictest0-21swapper/121:05:151
1488799210,3cyclictest131rcu_preempt00:35:131
1487899212,8cyclictest19919-21ntp_192.168.11421:55:230
1487899211,9cyclictest16030-21kthreadcore22:30:210
1487899211,6cyclictest30816-21fschecks_time20:00:150
14878992113,6cyclictest6023-21ssh00:30:310
1487899211,15cyclictest8774-21ssh22:25:260
1487899211,15cyclictest8774-21ssh22:25:260
1490599208,9cyclictest0-21swapper/323:55:223
1490599204,10cyclictest0-21swapper/300:20:113
1490599202,12cyclictest0-21swapper/322:10:133
1490599202,11cyclictest0-21swapper/319:50:133
14905992014,4cyclictest0-21swapper/322:41:213
14905992011,7cyclictest0-21swapper/322:37:273
14905992011,4cyclictest0-21swapper/322:46:223
1490599201,13cyclictest0-21swapper/321:30:143
1488799205,3cyclictest131rcu_preempt19:50:271
1488799204,5cyclictest21-21ksoftirqd/122:15:241
14887992014,4cyclictest0-21swapper/120:30:141
14887992014,4cyclictest0-21swapper/120:25:141
14887992014,4cyclictest0-21swapper/119:15:141
14887992012,6cyclictest0-21swapper/120:15:231
14887992012,5cyclictest21-21ksoftirqd/119:45:131
1487899209,3cyclictest131rcu_preempt22:20:280
1487899202,15cyclictest21990-21ssh21:40:120
14878992013,5cyclictest9207-21ssh22:08:440
1487899201,16cyclictest21158-21ls19:50:140
1487899201,14cyclictest11370-21ls19:40:010
1490599199,8cyclictest0-21swapper/300:15:333
1490599195,11cyclictest35966irq/58-eth019:45:143
1490599192,9cyclictest35966irq/58-eth020:30:123
14905991911,6cyclictest0-21swapper/323:20:213
1490599191,13cyclictest0-21swapper/320:45:283
14905991910,6cyclictest0-21swapper/322:21:473
1488799195,3cyclictest131rcu_preempt20:05:221
14887991911,5cyclictest0-21swapper/120:55:271
1488799190,12cyclictest1943-21cpuspeed19:30:141
1487899199,7cyclictest20037-21uptime00:20:280
1487899191,5cyclictest0-21swapper/023:01:150
1487899191,5cyclictest0-21swapper/023:01:150
14878991912,5cyclictest11658-21ssh23:59:240
1490599189,6cyclictest0-21swapper/322:01:203
1490599189,4cyclictest0-21swapper/322:16:363
1490599188,4cyclictest35966irq/58-eth021:48:333
1490599185,10cyclictest0-21swapper/300:35:023
1490599182,11cyclictest0-21swapper/321:10:143
1490599182,10cyclictest0-21swapper/320:50:143
1490599182,10cyclictest0-21swapper/320:10:153
1490599182,10cyclictest0-21swapper/320:10:153
1490599181,11cyclictest35966irq/58-eth019:30:133
14905991810,5cyclictest35966irq/58-eth023:42:003
14905991810,5cyclictest35966irq/58-eth023:41:593
1488799185,3cyclictest131rcu_preempt19:20:191
14887991813,3cyclictest0-21swapper/120:45:241
14887991812,4cyclictest0-21swapper/120:40:151
1487899189,6cyclictest7566-21cron20:10:010
1487899188,8cyclictest31643-21ntp_192.168.11421:05:240
1487899188,8cyclictest24388-21cron19:20:020
1487899182,14cyclictest21025-21kthreadcore20:55:200
1487899181,7cyclictest24788-21/usr/sbin/munin00:25:270
1487899181,7cyclictest24788-21/usr/sbin/munin00:25:270
14878991811,5cyclictest12039-21ssh00:15:490
1487899181,14cyclictest11483-21find19:40:120
1490599179,6cyclictest0-21swapper/323:50:293
1490599179,6cyclictest0-21swapper/323:08:313
1490599179,6cyclictest0-21swapper/321:43:533
1490599179,6cyclictest0-21swapper/300:25:333
1490599179,6cyclictest0-21swapper/300:25:333
1490599179,6cyclictest0-21swapper/300:08:573
1490599178,7cyclictest0-21swapper/322:59:263
1490599172,9cyclictest0-21swapper/320:20:113
14905991711,4cyclictest0-21swapper/321:23:393
14905991710,3cyclictest0-21swapper/300:37:283
1488799176,8cyclictest2024-21sed20:00:281
1487899171,7cyclictest2020-21fschecks_count19:30:160
1487899171,5cyclictest13639-21latency_hist20:50:020
1490599162,9cyclictest0-21swapper/321:00:133
1490599161,11cyclictest35966irq/58-eth019:15:133
1490599161,10cyclictest0-21swapper/319:55:323
1490599161,10cyclictest0-21swapper/319:30:023
1488799164,9cyclictest14899-21latency_hist19:10:011
1487899168,5cyclictest14090-21kthreadcore20:15:200
1487899167,7cyclictest29155-21users19:20:300
1487899162,2cyclictest131rcu_preempt20:35:260
1487899161,6cyclictest24843-21kthreadcore20:25:210
1490599152,10cyclictest0-21swapper/321:05:143
1490599152,10cyclictest0-21swapper/320:40:143
1490599151,11cyclictest0-21swapper/319:35:153
1490599151,11cyclictest0-21swapper/319:20:203
1487899151,3cyclictest131rcu_preempt19:45:170
1490599141,9cyclictest0-21swapper/320:55:173
1487899141,3cyclictest131rcu_preempt19:10:010
1490599138,3cyclictest0-21swapper/320:05:283
1490599136,5cyclictest0-21swapper/319:40:223
1490599132,9cyclictest0-21swapper/319:10:143
1490599124,4cyclictest0-21swapper/320:15:143
1490599112,6cyclictest0-21swapper/320:25:143
1490599101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional