You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 05:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Jan 05, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
527599632,44cyclictest9688-21ssh22:50:111
527599481,6cyclictest18647-21ssh22:55:101
527599481,6cyclictest18647-21ssh22:55:101
5284994612,11cyclictest633-21Async12:11:532
528499454,16cyclictest18872-21ssh21:25:132
527599402,7cyclictest5093-21apt-get00:00:151
529299370,34cyclictest0-21swapper/323:40:143
528499377,3cyclictest12142-21ssh21:55:312
528499377,3cyclictest12142-21ssh21:55:312
528499371,6cyclictest1693-21awk19:40:012
527599372,3cyclictest131rcu_preempt22:35:141
528499361,8cyclictest28115-21ntp_192.168.11419:30:252
528499361,13cyclictest26460-21tail20:35:292
528499361,12cyclictest15124-21kthreadcore00:05:212
528499360,10cyclictest21235-21/usr/sbin/munin20:00:252
527599361,13cyclictest26306-21df22:05:141
527599360,8cyclictest8680-21ls19:10:211
527599360,6cyclictest26046-21apt-get20:05:131
529299351,26cyclictest28366irq/58-eth022:55:103
529299351,26cyclictest28366irq/58-eth022:55:103
528499356,7cyclictest25462-21ssh23:35:062
528499351,9cyclictest24260-21/usr/sbin/munin22:40:262
528499351,9cyclictest14556-21grep21:00:172
528499351,8cyclictest16341-21kthreadcore22:35:212
528499351,6cyclictest23586-21ls19:25:322
528499351,5cyclictest1194666kworker/u8:100:32:482
528499351,5cyclictest1194666kworker/u8:100:32:482
528499351,13cyclictest24539-21kthreadcore23:15:212
528499351,12cyclictest19930-21fschecks_time23:50:172
528499350,9cyclictest22941-21cat21:05:332
527599351,9cyclictest10288-21kthreadcore00:20:231
527599351,10cyclictest22049-21cut21:05:271
528499345,6cyclictest15249-21cut21:40:212
528499341,9cyclictest3520-21cut21:15:232
528499341,9cyclictest21460-21tr22:20:192
528499341,8cyclictest11588-21/usr/sbin/munin19:50:282
528499341,10cyclictest6006-21apt-key22:30:022
528499340,9cyclictest5494-21grep19:40:282
528499340,9cyclictest10124-21kthreadcore20:20:222
528499340,10cyclictest31864-21diskstats00:15:162
527599347,24cyclictest3053-21kthreadcore23:40:211
527599341,9cyclictest14423-21cut19:50:271
527599341,8cyclictest29623-21cut21:30:211
527599341,8cyclictest2815-21grep23:20:311
527599341,8cyclictest20299-21ssh22:00:231
527599340,8cyclictest16491-21df_inode19:55:161
5292993317,8cyclictest28366irq/58-eth021:35:133
529299331,28cyclictest0-21swapper/322:20:143
528499336,4cyclictest8631-21kthreadcore23:25:212
528499332,12cyclictest9586-21/usr/sbin/munin20:55:272
528499331,9cyclictest9443-21cut22:30:232
528499331,8cyclictest28109-21kernelversion20:40:202
528499331,8cyclictest22834-21/usr/sbin/munin00:10:152
528499331,5cyclictest19713-21cat19:25:022
528499331,5cyclictest16603-21/usr/sbin/munin23:30:312
528499331,5cyclictest10321-21ssh22:50:172
528499331,14cyclictest25704-21awk00:30:012
528499331,10cyclictest22476-21latency_hist20:35:012
528499330,9cyclictest32301-21cut20:10:222
528499330,9cyclictest31817-21timerwakeupswit21:30:302
528499330,8cyclictest5872-21irqcore19:10:172
528499330,5cyclictest6952-21fschecks_time19:45:152
528499330,11cyclictest31848-21grep23:00:312
527599339,20cyclictest29791-21diskstats22:25:171
527599331,9cyclictest6749-21kthreadcore20:50:211
527599331,8cyclictest32354-21kthreadcore19:35:231
527599331,8cyclictest23800-21cut21:45:201
527599331,8cyclictest12648-21kthreadcore23:45:211
527599331,7cyclictest3018-21kthreadcore19:40:221
527599331,7cyclictest27717-21/usr/sbin/munin23:00:141
527599331,6cyclictest31857-21kthreadcore20:10:211
527599331,5cyclictest22275-21cpuspeed23:15:141
527599331,10cyclictest851-21sshd00:33:531
527599331,10cyclictest851-21sshd00:33:531
527599331,10cyclictest26626-21grep21:10:221
527599330,5cyclictest0-21swapper/100:28:301
527599330,10cyclictest17309-21wc23:10:341
529299322,17cyclictest28366irq/58-eth023:00:133
529299321,25cyclictest0-21swapper/321:25:123
529299320,28cyclictest0-21swapper/321:55:153
529299320,28cyclictest0-21swapper/321:55:143
528499321,9cyclictest5321-21kthreadcore22:10:242
528499321,9cyclictest4613-21if_eth123:05:192
528499321,9cyclictest31601-21ssh22:06:092
528499321,9cyclictest3026-21irqcore00:35:212
528499321,9cyclictest0-21swapper/221:36:542
528499321,7cyclictest5358-21cut00:00:192
528499321,14cyclictest10212-21/usr/sbin/munin21:20:182
528499321,11cyclictest16394-21acpi19:55:132
528499321,10cyclictest18215-21ssh23:10:382
528499320,9cyclictest6183-21cut20:15:242
528499320,10cyclictest4610-21grep20:50:022
527599321,9cyclictest3636-21kthreadcore00:35:211
527599321,9cyclictest15871-21ssh21:40:221
527599321,9cyclictest14298-21cut20:25:201
527599321,8cyclictest9810-21/usr/sbin/munin21:20:191
527599321,8cyclictest32180-21sed21:50:191
527599321,8cyclictest25114-21grep20:00:301
527599321,8cyclictest20635-21/usr/sbin/munin22:20:171
527599321,8cyclictest18991-21diskstats21:25:161
527599321,7cyclictest913-21kthreadcore20:45:171
527599321,7cyclictest27798-21cut23:35:221
527599321,7cyclictest23800-21ssh23:50:311
527599321,7cyclictest15596-21irqrtprio19:20:201
527599321,6cyclictest4404-21/usr/sbin/munin21:35:201
527599321,6cyclictest31938-21grep00:15:171
527599321,5cyclictest6730-21/usr/sbin/munin22:30:191
527599321,4cyclictest4241-21ssh23:05:141
527599320,8cyclictest843-21diskstats21:15:161
529299311,26cyclictest0-21swapper/322:30:053
529299311,18cyclictest28366irq/58-eth021:10:133
528499318,5cyclictest6930-21ssh23:40:392
528499311,9cyclictest0-21swapper/223:58:402
528499311,6cyclictest13082-21df_inode20:25:152
528499311,13cyclictest9557-21kthreadcore00:20:222
527599313,7cyclictest10129-21kthreadcore21:55:231
527599313,7cyclictest10129-21kthreadcore21:55:231
527599311,9cyclictest8276-21ps23:25:211
527599311,9cyclictest10091-21irqcore20:55:181
527599311,8cyclictest17527-21sed00:05:241
527599311,8cyclictest15186-21kthreadcore21:00:221
527599311,7cyclictest32702-21ssh22:44:511
527599311,7cyclictest23836-21diskmemload23:30:411
527599311,7cyclictest23711-21cut00:10:191
527599311,6cyclictest10341-21find19:45:261
527599311,5cyclictest17919-21diskstats20:30:161
529299301,23cyclictest0-21swapper/319:25:163
529299300,25cyclictest0-21swapper/322:45:143
529299300,25cyclictest0-21swapper/322:45:143
529299300,23cyclictest0-21swapper/320:55:133
529299300,23cyclictest0-21swapper/320:25:123
528499309,3cyclictest22609-21switchtime22:55:272
528499309,3cyclictest22609-21switchtime22:55:272
528499309,2cyclictest26028-21/usr/sbin/munin20:05:192
528499305,6cyclictest30735-21ssh21:12:292
528499301,8cyclictest20393-21kthreadcore22:00:222
528499301,7cyclictest31969-21tail21:50:192
528499301,10cyclictest2663-21timerandwakeup23:20:312
528499300,9cyclictest5115-21grep20:50:172
528499300,6cyclictest10727-21ssh23:45:112
528499300,5cyclictest6040-21rm22:46:012
528499300,5cyclictest6040-21rm22:46:012
528499300,5cyclictest18108-21ssh22:17:212
527599301,9cyclictest21844-21cut19:25:231
527599301,9cyclictest12416-21sed22:15:191
527599301,7cyclictest7870-21users20:15:351
527599301,6cyclictest12606-21cut19:15:241
527599301,5cyclictest32181-21ntp_192.168.11423:55:261
527599300,9cyclictest27576-21fschecks_count20:40:171
527599300,6cyclictest1013-21fschecks_time22:45:171
527599300,6cyclictest1013-21fschecks_time22:45:171
526799304,22cyclictest1961-21ssh23:40:150
5292992918,9cyclictest28466irq/59-eth023:12:293
5292992917,10cyclictest28466irq/59-eth023:09:453
529299291,25cyclictest0-21swapper/323:49:273
529299291,23cyclictest0-21swapper/319:15:153
529299291,22cyclictest0-21swapper/322:10:153
529299290,25cyclictest0-21swapper/322:40:163
529299290,22cyclictest0-21swapper/321:20:153
529299290,22cyclictest0-21swapper/300:00:163
527599291,6cyclictest3007-21tune2fs22:10:181
527599291,6cyclictest23174-21irqrtprio20:35:181
527599291,4cyclictest0-21swapper/119:30:171
5267992925,2cyclictest0-21swapper/000:05:130
529299280,25cyclictest0-21swapper/322:15:113
528499281,8cyclictest13810-21mailstats19:15:302
528499280,9cyclictest5304-21latency_hist19:10:022
527599281,12cyclictest8156-21apt-get20:20:141
526799283,21cyclictest26215-21ssh22:05:130
526799282,21cyclictest20690-21ssh22:20:140
529299271,22cyclictest0-21swapper/322:25:143
529299270,22cyclictest0-21swapper/300:30:143
529299270,22cyclictest0-21swapper/300:30:143
526799273,20cyclictest19712-21ssh23:50:130
526799272,22cyclictest26111-21ssh00:30:150
526799272,22cyclictest26111-21ssh00:30:140
5267992714,7cyclictest131rcu_preempt20:20:140
526799271,23cyclictest9845-21ssh21:20:140
529299261,21cyclictest0-21swapper/300:25:133
529299260,23cyclictest0-21swapper/323:55:133
529299260,22cyclictest0-21swapper/323:20:143
526799263,20cyclictest28535-21ssh23:55:140
526799262,20cyclictest22218-21sh23:15:140
5267992614,10cyclictest12-21ksoftirqd/023:35:130
5267992612,11cyclictest4465-21cron20:50:010
529299251,20cyclictest0-21swapper/323:35:133
529299251,20cyclictest0-21swapper/322:00:163
529299251,20cyclictest0-21swapper/300:20:133
529299251,18cyclictest0-21swapper/322:35:133
529299250,22cyclictest0-21swapper/320:40:133
529299250,21cyclictest0-21swapper/321:15:133
529299250,20cyclictest0-21swapper/320:05:133
529299250,20cyclictest0-21swapper/319:35:133
527599250,6cyclictest5252-21cyclictest19:10:011
526799252,19cyclictest12-21ksoftirqd/022:25:140
526799252,15cyclictest839-21munin-node19:20:140
526799251,19cyclictest4441-21ssh21:35:140
529299241,20cyclictest0-21swapper/322:52:183
529299241,18cyclictest0-21swapper/319:30:143
529299240,19cyclictest0-21swapper/300:10:133
526799244,9cyclictest0-21swapper/019:50:190
526799242,19cyclictest23836-21diskmemload23:45:140
526799242,18cyclictest820-21ssh22:45:150
526799242,18cyclictest820-21ssh22:45:140
526799242,18cyclictest1529166ssh22:35:140
526799242,17cyclictest31592-21ssh21:50:150
526799242,17cyclictest13451-21apt-get21:40:130
526799241,19cyclictest20553-21ls23:30:280
5267992411,11cyclictest12-21ksoftirqd/019:30:140
529299231,19cyclictest0-21swapper/320:30:183
529299231,19cyclictest0-21swapper/320:20:293
529299231,19cyclictest0-21swapper/319:55:323
529299231,19cyclictest0-21swapper/319:50:343
529299231,19cyclictest0-21swapper/319:10:123
529299231,18cyclictest0-21swapper/320:50:173
529299230,20cyclictest0-21swapper/321:50:133
529299230,20cyclictest0-21swapper/300:17:273
529299230,20cyclictest0-21swapper/300:05:133
529299230,19cyclictest0-21swapper/321:40:133
529299230,19cyclictest0-21swapper/320:45:113
526799239,9cyclictest0-21swapper/021:00:140
526799233,17cyclictest17199-21ssh22:00:140
526799232,15cyclictest839-21munin-node21:05:190
5267992312,5cyclictest0-21swapper/020:00:140
529299221,18cyclictest0-21swapper/323:50:343
529299221,18cyclictest0-21swapper/321:45:303
529299221,18cyclictest0-21swapper/300:35:303
529299220,19cyclictest0-21swapper/323:25:143
529299220,19cyclictest0-21swapper/322:05:113
529299220,19cyclictest0-21swapper/321:30:163
529299220,18cyclictest0-21swapper/320:35:123
526799224,4cyclictest131rcu_preempt22:55:100
526799224,4cyclictest131rcu_preempt22:55:090
526799223,16cyclictest2829-21ssh22:10:160
526799222,17cyclictest731-21ssh21:15:140
526799222,14cyclictest839-21munin-node20:10:160
5267992212,6cyclictest7031-21usage_EthcTask00:35:330
526799221,18cyclictest6360-21ssh22:30:140
526799221,10cyclictest22719-21cut21:25:290
529299211,17cyclictest0-21swapper/323:15:123
529299211,17cyclictest0-21swapper/320:10:313
529299211,16cyclictest0-21swapper/323:30:113
529299210,18cyclictest0-21swapper/320:00:133
529299210,18cyclictest0-21swapper/319:45:093
529299210,18cyclictest0-21swapper/319:40:143
529299210,18cyclictest0-21swapper/319:20:143
526799219,8cyclictest12-21ksoftirqd/000:10:130
526799219,4cyclictest131rcu_preempt20:05:120
526799213,13cyclictest8423-21sed23:05:310
526799213,13cyclictest8136-21ssh21:55:150
526799213,13cyclictest8136-21ssh21:55:150
526799213,13cyclictest0-21swapper/000:00:140
526799213,11cyclictest17213-21ssh00:25:130
526799212,16cyclictest9677-21apt-get22:50:100
5267992113,5cyclictest0-21swapper/000:20:200
5267992111,7cyclictest19864-21sshd22:19:320
526799211,15cyclictest32657-21ssh23:00:350
526799211,14cyclictest4233-21ssh00:15:330
526799211,14cyclictest27790-21sh21:30:130
529299201,16cyclictest0-21swapper/321:00:143
529299201,16cyclictest0-21swapper/320:15:153
529299200,18cyclictest0-21swapper/321:05:163
526799202,13cyclictest24253-21sh21:10:140
5267992011,5cyclictest23255-21kthreadcore21:45:210
526799201,15cyclictest19682-21ls19:55:240
526799193,12cyclictest12-21ksoftirqd/019:40:150
526799192,14cyclictest23324-21ntpq19:25:280
526799191,6cyclictest2324-21ntp_192.168.11423:20:260
5267991910,6cyclictest11826-21unixbench_multi23:25:320
526799189,7cyclictest24284-21apt-get22:40:150
526799185,9cyclictest13275-21diskmemload23:10:190
526799184,8cyclictest0-21swapper/019:45:110
526799183,11cyclictest9970-21irqcore20:55:180
5267991811,5cyclictest6071-21tail19:10:170
526799181,15cyclictest30508-21kthreadcore20:40:220
526799181,13cyclictest3374-21diskstats20:15:190
526799181,13cyclictest27210-21munin-run20:40:020
5267991810,5cyclictest0-21swapper/020:25:290
526799175,9cyclictest8659-21grep20:50:300
526799172,11cyclictest534-21ntp_states19:35:270
5267991611,3cyclictest18084-21if_err_eth020:30:180
529299151,11cyclictest0-21swapper/319:10:023
5267991510,3cyclictest10880-21kernelversion19:15:210
526799141,9cyclictest5268-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional