You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 13:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Mar 20, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3258899673,48cyclictest30088-21ssh22:45:120
3260999632,40cyclictest0-21swapper/200:05:112
3260099636,48cyclictest6744-21ssh22:50:121
3260099561,42cyclictest3040-21ssh00:35:121
3258899561,43cyclictest20973-21ssh22:40:120
32619995438,11cyclictest28366irq/58-eth022:40:123
3260099545,39cyclictest21870-21ssh22:05:121
3260999531,44cyclictest30965-21apt-get22:10:112
3260999520,48cyclictest4770-21apt-get21:20:142
3260999512,44cyclictest0-21swapper/223:25:122
3260999492,42cyclictest0-21swapper/200:20:122
3260999491,42cyclictest29369-21apt-get19:40:122
3260999482,35cyclictest0-21swapper/220:20:132
3260999481,38cyclictest0-21swapper/223:40:122
3260999471,40cyclictest0-21swapper/223:10:122
3260999463,32cyclictest0-21swapper/223:35:132
3260999462,32cyclictest0-21swapper/223:50:122
3260999462,32cyclictest0-21swapper/223:50:122
3258899451,35cyclictest25959-21ssh22:25:130
32600994414,25cyclictest21-21ksoftirqd/123:50:121
32600994414,25cyclictest21-21ksoftirqd/123:50:121
3258899442,36cyclictest4780-21ssh21:20:130
3260999432,37cyclictest0-21swapper/222:00:122
3260999431,36cyclictest0-21swapper/200:01:422
32619994210,22cyclictest28366irq/58-eth021:20:143
3260999421,37cyclictest0-21swapper/222:55:132
32600994234,3cyclictest27584-21ssh23:55:131
3260099421,6cyclictest131rcu_preempt21:15:121
3260999411,35cyclictest13900-21apt-get21:25:142
3260999411,33cyclictest11757-21apt-get22:35:132
3260999410,33cyclictest15566-21ssh22:19:022
3260999410,33cyclictest15566-21ssh22:19:022
3260099411,32cyclictest27005-21ssh21:50:121
3260999401,35cyclictest9166-21apt-get21:00:112
3260999401,34cyclictest0-21swapper/223:30:132
3260999400,32cyclictest25411-21memory22:05:232
3260999400,25cyclictest3483-21ssh21:19:012
3260999400,14cyclictest4548-21ssh00:16:102
3261999391,21cyclictest28366irq/58-eth019:30:103
3260999394,14cyclictest27-21ksoftirqd/221:05:132
3260999391,34cyclictest23078-21apt-get21:30:132
32600993921,11cyclictest666-21ntpd20:55:131
3261999382,21cyclictest28366irq/58-eth022:15:123
3261999382,21cyclictest28366irq/58-eth022:15:123
3260999381,24cyclictest12083-21ssh21:59:372
3260999380,31cyclictest23785-21sh00:27:252
3260999380,23cyclictest0-21swapper/223:04:212
3260099385,25cyclictest21-21ksoftirqd/122:00:121
3261999373,18cyclictest28366irq/58-eth000:00:143
3260999372,32cyclictest19427-21cut21:45:202
3260999370,26cyclictest9175-21sh00:36:572
3260099371,20cyclictest16829-21ssh22:20:131
3260999361,32cyclictest0-21swapper/220:45:132
3260999361,29cyclictest0-21swapper/221:50:122
3260999361,25cyclictest531-21usb-storage19:20:112
3258899362,27cyclictest18543-21diskmemload21:35:120
3258899361,27cyclictest2652-21ssh22:30:120
3260999351,31cyclictest8017-21/usr/sbin/munin20:55:282
3260999351,21cyclictest26522-21ps20:10:192
3260999350,31cyclictest4910-21users19:10:302
3260999343,23cyclictest11355-21rm22:34:592
3260999341,29cyclictest5044-21memory19:45:212
3260999341,26cyclictest25956-21apt-get22:25:142
3260999341,24cyclictest924-21snmpd00:10:042
3260999341,23cyclictest24854-21ssh22:24:112
3260999341,20cyclictest0-21swapper/223:23:252
3260999341,19cyclictest0-21swapper/200:31:002
3260999340,30cyclictest0-21swapper/220:15:132
3260999340,26cyclictest27324-21sh22:42:152
3260999340,22cyclictest924-21snmpd21:10:072
3260999340,21cyclictest11405-21mailstats20:25:282
3260099342,23cyclictest26379-21ssh00:30:131
32619993320,9cyclictest28366irq/58-eth022:05:333
3260999331,29cyclictest2944-21ntp_states20:50:262
3260999331,28cyclictest6353-21cat23:05:302
3260999331,28cyclictest12750-21diskstats20:30:162
3260999331,26cyclictest9541-21ssh23:45:142
3260999330,29cyclictest5528-21ssh21:36:352
3260999330,29cyclictest0-21swapper/220:05:122
3260999330,22cyclictest19703-21ssh23:15:092
3260999330,22cyclictest14122-21ssh21:41:012
3260999330,22cyclictest14122-21ssh21:41:012
3260999330,19cyclictest4732-21ssh22:48:062
3260099333,27cyclictest24857-21ssh23:00:131
32600993315,14cyclictest21-21ksoftirqd/123:35:131
3258899331,25cyclictest17302-21ssh00:25:120
3260999321,28cyclictest924-21snmpd19:25:142
3260999321,27cyclictest16202-21apt-get20:00:152
3260999321,18cyclictest0-21swapper/223:57:322
3260999320,29cyclictest924-21snmpd19:19:452
3260099322,21cyclictest17921-21ssh21:45:131
32600993213,9cyclictest32576-21cyclictest23:40:131
3261999311,18cyclictest28366irq/58-eth023:45:133
3260999311,20cyclictest10780-21unixbench_multi22:50:282
32600993116,8cyclictest0-21swapper/120:35:121
3258899319,4cyclictest131rcu_preempt22:55:120
32588993122,4cyclictest531-21usb-storage23:05:110
3260999301,26cyclictest28273-21unixbench_multi19:35:262
3260999300,27cyclictest6712-21diskstats19:50:142
3260999300,27cyclictest26821-21sshd20:41:382
3260999300,26cyclictest0-21swapper/220:35:112
3260999300,22cyclictest24221-21sshd19:30:382
3258899304,21cyclictest12-21ksoftirqd/021:30:140
32588993012,8cyclictest12-21ksoftirqd/019:20:110
3260999290,26cyclictest15928-21sshd19:55:382
32600992915,4cyclictest531-21usb-storage19:50:111
3258899292,20cyclictest629-21ssh00:32:420
3258899291,6cyclictest12-21ksoftirqd/020:40:120
3261999283,19cyclictest0-21swapper/300:16:423
3261999283,18cyclictest28366irq/58-eth021:35:123
3261999283,16cyclictest0-21swapper/300:08:423
3258899286,15cyclictest29991-1kworker/0:2H19:40:050
3258899282,20cyclictest29668-21ssh00:13:200
3260099272,18cyclictest19900-21ssh23:15:151
3260099272,17cyclictest11769-21ssh22:35:121
32600992715,9cyclictest7787-21cron00:20:011
3258899274,3cyclictest131rcu_preempt23:45:140
3258899271,20cyclictest18947-21ssh21:10:140
3258899269,12cyclictest12-21ksoftirqd/000:02:420
3258899263,17cyclictest14722-21ssh23:30:130
32588992610,10cyclictest633-21Async12:11:530
3261999251,12cyclictest28366irq/58-eth022:45:123
3260099251,19cyclictest13935-21sh21:25:141
3258899258,12cyclictest12-21ksoftirqd/023:59:420
3258899256,6cyclictest12-21ksoftirqd/023:40:120
3261999242,12cyclictest28366irq/58-eth023:05:113
32619992412,6cyclictest28366irq/58-eth023:00:133
3260099244,14cyclictest22392-21sh00:10:131
3260099243,11cyclictest10064-21apt-get19:20:111
32600992415,3cyclictest0-21swapper/100:25:131
3258899243,16cyclictest0-21swapper/000:39:420
3261999233,11cyclictest0-21swapper/321:10:123
32619992314,6cyclictest0-21swapper/323:20:163
32619992313,7cyclictest28366irq/58-eth022:35:133
3260099232,16cyclictest8495-21ssh21:20:271
3260099232,16cyclictest13993-21apt-get21:05:121
3260099232,11cyclictest32576-21cyclictest19:25:131
3258899238,10cyclictest16644-21ssh00:24:420
32588992317,4cyclictest131rcu_preempt19:30:110
32588992313,7cyclictest7627-21cron20:25:010
32588992312,8cyclictest10481-21users23:25:310
32588992312,3cyclictest0-21swapper/021:57:130
3261999223,10cyclictest28366irq/58-eth020:00:143
32619992214,5cyclictest28366irq/58-eth022:55:123
3260099225,9cyclictest0-21swapper/122:25:121
3260099225,12cyclictest0-21swapper/121:30:141
3260099222,15cyclictest24477-21apt-get19:35:111
3260099222,15cyclictest20994-21ssh22:40:141
32600992215,4cyclictest9592-21cpu23:45:141
3260099221,14cyclictest31555-21ssh22:10:181
3260099221,14cyclictest26174-21tr20:10:181
3258899228,8cyclictest0-21swapper/023:21:310
3258899228,5cyclictest12-21ksoftirqd/021:40:180
3258899228,5cyclictest12-21ksoftirqd/021:40:180
3258899226,6cyclictest12-21ksoftirqd/022:11:030
3258899226,10cyclictest18543-21diskmemload00:16:420
32588992215,4cyclictest12-21ksoftirqd/019:55:200
3258899221,18cyclictest30791-21cut20:15:170
3258899221,14cyclictest6903-21cut22:50:150
32619992113,5cyclictest28366irq/58-eth021:55:143
3260099219,9cyclictest0-21swapper/120:50:131
3260099212,17cyclictest20866-21ssh00:08:421
3260099211,14cyclictest7707-21ssh22:15:131
3260099211,14cyclictest7707-21ssh22:15:131
3260099211,14cyclictest10640-21ssh23:10:131
32588992110,3cyclictest12-21ksoftirqd/023:38:120
32588992110,3cyclictest12-21ksoftirqd/023:11:210
32588992110,3cyclictest12-21ksoftirqd/021:52:040
3261999209,7cyclictest28366irq/58-eth023:10:133
3261999208,7cyclictest28366irq/58-eth000:25:143
32619992015,3cyclictest28366irq/58-eth023:37:013
32619992011,6cyclictest0-21swapper/323:25:293
3261999201,14cyclictest0-21swapper/321:45:013
3261999201,14cyclictest0-21swapper/321:45:013
3260099201,3cyclictest131rcu_preempt22:55:151
32600992010,7cyclictest0-21swapper/121:00:111
3258899209,8cyclictest24571-21diskstats19:35:140
3258899209,3cyclictest12-21ksoftirqd/023:50:120
3258899209,3cyclictest12-21ksoftirqd/023:50:120
3258899208,3cyclictest12-21ksoftirqd/022:15:530
3258899208,3cyclictest12-21ksoftirqd/022:15:530
3258899208,3cyclictest12-21ksoftirqd/021:10:020
32588992014,4cyclictest0-21swapper/021:45:120
32588992014,4cyclictest0-21swapper/021:25:130
32588992013,3cyclictest12-21ksoftirqd/023:00:220
3258899201,15cyclictest11747-21find22:35:110
3258899201,14cyclictest18517-21ls19:25:280
3258899200,14cyclictest27184-21cut20:45:150
3261999199,8cyclictest0-21swapper/322:13:493
3261999199,3cyclictest0-21swapper/322:04:593
3261999193,11cyclictest0-21swapper/300:32:423
3261999193,11cyclictest0-21swapper/300:23:253
3261999192,11cyclictest0-21swapper/321:05:133
3261999191,6cyclictest0-21swapper/300:35:423
32619991913,4cyclictest0-21swapper/321:26:113
32619991913,4cyclictest0-21swapper/300:10:253
3261999191,13cyclictest0-21swapper/320:15:283
3261999191,13cyclictest0-21swapper/319:15:023
3261999191,11cyclictest0-21swapper/321:05:013
3261999191,11cyclictest0-21swapper/320:15:023
32619991910,7cyclictest0-21swapper/322:50:123
3260099197,9cyclictest24903-21grep20:05:271
3260099194,12cyclictest4941-21cron19:15:011
3260099192,12cyclictest5622-21ssh23:25:121
3260099192,10cyclictest8075-21ls00:00:291
3260099191,8cyclictest24245-21latency_hist19:35:011
3260099191,15cyclictest6546-21latency_hist22:50:011
3260099191,15cyclictest22264-21apt-get20:40:131
3260099191,14cyclictest1784-21sort21:35:211
32600991911,4cyclictest16872-21kthreadcore20:00:201
3260099191,13cyclictest3861-21ssh21:55:161
32600991910,6cyclictest2631-21apt-get22:30:121
32600991910,5cyclictest9042-21kthreadcore00:20:181
3260099190,14cyclictest7991-21cut20:25:161
3258899198,8cyclictest32767-21apt-get19:10:120
3258899198,3cyclictest12-21ksoftirqd/022:05:130
3258899197,8cyclictest10011-21latency_hist19:20:020
3258899194,5cyclictest12-21ksoftirqd/021:18:510
3258899193,13cyclictest12-21ksoftirqd/020:50:130
3258899192,14cyclictest15978-21missed_timers20:30:230
32588991913,3cyclictest12-21ksoftirqd/020:00:210
32588991911,4cyclictest131rcu_preempt23:15:130
32588991910,5cyclictest12-21ksoftirqd/021:00:010
3261999189,6cyclictest0-21swapper/323:42:203
3261999188,8cyclictest33-21ksoftirqd/321:33:373
3261999188,6cyclictest28366irq/58-eth021:45:133
3261999187,8cyclictest0-21swapper/323:18:173
3261999187,7cyclictest0-21swapper/322:25:143
3261999182,10cyclictest0-21swapper/320:45:133
3261999182,10cyclictest0-21swapper/319:40:053
32619991811,5cyclictest0-21swapper/323:55:213
3261999181,10cyclictest0-21swapper/319:50:053
32619991810,6cyclictest0-21swapper/323:32:243
32619991810,6cyclictest0-21swapper/322:24:263
3260099183,11cyclictest28817-21ssh23:20:041
3260099182,12cyclictest14825-21df_inode23:30:141
3260099181,9cyclictest12746-21kthreadcore19:55:181
32600991813,2cyclictest0-21swapper/119:45:131
3260099181,15cyclictest12895-21ls20:30:171
3260099181,14cyclictest3024-21cat20:20:021
3260099180,15cyclictest19141-21uniq21:10:161
3258899185,10cyclictest21145-21fschecks_count20:05:150
3258899183,11cyclictest30544-21cat20:15:020
3258899181,4cyclictest0-21swapper/022:20:220
32588991812,4cyclictest0-21swapper/019:50:020
32588991812,3cyclictest12-21ksoftirqd/021:01:380
32588991811,4cyclictest12-21ksoftirqd/020:35:150
3258899181,13cyclictest14806-21tr22:00:210
3258899181,11cyclictest0-21swapper/020:25:120
3261999179,6cyclictest0-21swapper/322:33:333
3261999179,6cyclictest0-21swapper/321:15:033
3261999177,8cyclictest0-21swapper/321:50:543
3261999171,11cyclictest0-21swapper/323:50:153
3261999171,11cyclictest0-21swapper/323:50:153
3260099172,12cyclictest1090-21apt-get23:05:051
3260099172,11cyclictest8962-21diskstats21:40:151
3260099172,11cyclictest8962-21diskstats21:40:151
3260099172,11cyclictest6312-21wc20:20:211
3260099172,11cyclictest27076-21apt-get20:45:111
3260099171,14cyclictest7533-21kthreadcore19:15:201
32600991710,4cyclictest0-21swapper/119:40:241
3258899172,11cyclictest1743-21acpi19:45:110
3261999162,10cyclictest0-21swapper/319:35:123
3260099162,11cyclictest32677-21latency_hist19:10:011
3261999151,9cyclictest0-21swapper/320:55:053
3261999151,7cyclictest0-21swapper/319:15:113
3261999151,10cyclictest33-21ksoftirqd/320:50:133
3261999149,3cyclictest321rcuc/320:26:383
3261999141,10cyclictest0-21swapper/320:30:143
3260999141,9cyclictest32644-21latency_hist19:10:012
3258899142,8cyclictest32576-21cyclictest19:10:010
3261999131,9cyclictest0-21swapper/319:55:123
3261999131,10cyclictest0-21swapper/319:25:163
3261999123,3cyclictest0-21swapper/319:20:113
3261999121,8cyclictest0-21swapper/320:05:153
3261999121,5cyclictest0-21swapper/320:35:273
3261999112,5cyclictest0-21swapper/320:40:143
3261999111,7cyclictest0-21swapper/319:45:193
3261999111,6cyclictest0-21swapper/320:20:183
3261999101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional