You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Fri May 15, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2368121780,6sleep32851699cyclictest08:08:363
1426721700,5sleep32851699cyclictest09:16:003
1361821050,1sleep00-21swapper/005:16:270
2499921040,5sleep32851699cyclictest08:49:393
63962950,2sleep30-21swapper/306:26:263
74312880,2sleep10-21swapper/107:10:121
195832880,3sleep10-21swapper/108:03:391
195832880,3sleep10-21swapper/108:03:391
28513998731,40cyclictest12211-21cron05:15:590
28513998731,40cyclictest12211-21cron05:15:590
2851399856,69cyclictest0-21swapper/009:21:100
28513997926,35cyclictest0-21swapper/009:18:490
28513997924,44cyclictest0-21swapper/008:28:050
2851399791,28cyclictest2456-21grep08:21:300
2851399782,23cyclictest5360-21apt-config06:26:000
28513997821,44cyclictest0-21swapper/009:46:230
28513997813,4cyclictest24876-21sshd09:28:040
28513997813,4cyclictest24876-21sshd09:28:040
2851399768,20cyclictest0-21swapper/007:21:090
28513997620,41cyclictest0-21swapper/008:11:170
2851399761,72cyclictest0-21swapper/007:46:000
2851399752,25cyclictest0-21swapper/008:31:120
28513997518,13cyclictest0-21swapper/007:54:260
2851399751,22cyclictest20959-21latency_hist08:06:000
2851399751,22cyclictest20959-21latency_hist08:06:000
2851399743,29cyclictest0-21swapper/007:26:100
2851399739,50cyclictest9-21ksoftirqd/004:51:090
28513997330,5cyclictest14928-21grep08:36:280
2851399717,14cyclictest0-21swapper/008:07:370
2851399717,11cyclictest7850irq/56-dwc_otg_06:36:120
28513997119,6cyclictest7850irq/56-dwc_otg_07:01:110
28513997119,6cyclictest7850irq/56-dwc_otg_07:01:110
2851399701,7cyclictest15299-21grep05:21:260
2851399701,19cyclictest0-21swapper/009:15:590
2851399701,19cyclictest0-21swapper/006:58:420
2851399701,18cyclictest24750-21diskmemload07:31:100
28513996922,6cyclictest25577-21systemd-journal05:46:130
2851399691,21cyclictest1926-21tune2fs04:46:150
2851399683,22cyclictest6503-21fschecks_time09:06:170
28513996817,13cyclictest0-21swapper/005:40:430
28513996815,10cyclictest0-21swapper/005:32:290
2851399681,4cyclictest7950irq/56-dwc_otg_09:04:460
2851399681,31cyclictest0-21swapper/007:06:120
2851399681,20cyclictest0-21swapper/008:56:250
2851399681,19cyclictest0-21swapper/007:37:320
2851399681,16cyclictest0-21swapper/009:36:140
320932670,5sleep12851499cyclictest09:36:271
2851399673,22cyclictest0-21swapper/007:20:590
2851399663,22cyclictest2638-21apt-get09:41:120
2851399662,22cyclictest0-21swapper/005:05:020
28513996614,13cyclictest7850irq/56-dwc_otg_06:31:110
2851399652,27cyclictest0-21swapper/009:35:120
2851399652,20cyclictest1028-21unixbench-2d06:11:260
2851399652,20cyclictest0-21swapper/008:45:120
2851399652,20cyclictest0-21swapper/008:45:120
2851399652,18cyclictest0-21swapper/008:48:010
2851399651,27cyclictest0-21swapper/004:36:030
2851399651,21cyclictest0-21swapper/008:00:230
139912650,2sleep20-21swapper/207:56:272
2851399642,8cyclictest20291-21ssh06:46:160
2851399642,23cyclictest0-21swapper/007:46:200
2851399632,47cyclictest0-21swapper/009:51:100
28513996318,11cyclictest0-21swapper/006:52:470
2851399622,14cyclictest29280-21rm08:54:530
271342620,8sleep1221ktimersoftd/104:26:181
2851699613,13cyclictest7750irq/56-dwc_otg08:16:113
281452603,50sleep30-21swapper/304:29:453
2851399595,12cyclictest0-21swapper/007:15:500
2851399592,19cyclictest0-21swapper/006:41:570
2851399581,18cyclictest908-21runrttasks06:05:320
2851399581,18cyclictest908-21runrttasks06:05:320
2851399581,16cyclictest7850irq/56-dwc_otg_04:45:590
2851399561,18cyclictest0-21swapper/008:19:390
248882560,1sleep10-21swapper/109:28:061
248882560,1sleep10-21swapper/109:28:061
63112550,5sleep22851599cyclictest07:47:012
2851399552,36cyclictest0-21swapper/005:06:270
28513995511,21cyclictest15805-21/usr/sbin/munin09:56:230
28514995426,4cyclictest26580-21if_wlan008:51:191
2851399532,28cyclictest28599-21wget04:31:120
2851399532,28cyclictest28599-21wget04:31:120
2851399531,35cyclictest21811-21fschecks_count05:41:140
2851499527,29cyclictest4696-21runrttasks07:46:101
2851399522,32cyclictest0-21swapper/006:16:250
279762523,12sleep00-21swapper/004:27:400
2851699514,20cyclictest6909-21ssh09:46:133
28513995023,15cyclictest0-21swapper/005:29:100
2851399501,27cyclictest0-21swapper/006:26:050
2851399494,27cyclictest0-21swapper/004:56:160
2851399482,30cyclictest0-21swapper/005:51:250
2851699478,25cyclictest0-21swapper/305:06:123
2851499475,7cyclictest7750irq/56-dwc_otg07:51:101
2851499472,10cyclictest7750irq/56-dwc_otg09:31:111
2851699468,23cyclictest0-21swapper/304:51:123
2851699467,25cyclictest0-21swapper/304:31:123
2851699467,25cyclictest0-21swapper/304:31:123
2851499468,25cyclictest0-21swapper/108:31:051
2851499462,7cyclictest7750irq/56-dwc_otg08:56:121
2851699451,16cyclictest9331-21ssh08:31:003
2851499452,5cyclictest0-21swapper/106:51:101
28516994411,4cyclictest0-21swapper/307:10:593
2851399441,27cyclictest0-21swapper/006:06:240
56942430,5sleep22851599cyclictest09:44:372
2851699437,22cyclictest0-21swapper/309:01:053
2851699435,24cyclictest8234-21ssh07:11:133
28515994337,4cyclictest29-21rcuc/208:41:102
28515994337,4cyclictest29-21rcuc/208:41:102
2851499432,8cyclictest6914-21ssh09:46:131
13702430,9sleep3381ktimersoftd/307:01:403
13702430,9sleep3381ktimersoftd/307:01:403
2851699427,25cyclictest0-21swapper/305:31:113
2851699427,23cyclictest0-21swapper/309:36:133
28516994212,12cyclictest0-21swapper/309:51:113
2851699421,20cyclictest13991-21fschecks_time08:36:163
2851599424,7cyclictest7850irq/56-dwc_otg_09:16:132
2851599422,6cyclictest0-21swapper/204:46:092
28515994210,7cyclictest22663-21tail05:41:252
2851399422,20cyclictest0-21swapper/005:56:270
2851699417,22cyclictest0-21swapper/306:16:123
2851699412,5cyclictest7750irq/56-dwc_otg09:56:143
28516994112,5cyclictest0-21swapper/307:57:083
2851599412,9cyclictest884450ssh07:51:132
160642410,1sleep30-21swapper/306:41:153
2851699406,20cyclictest917-21snmpd05:36:123
2851699405,21cyclictest22732-21ssh09:26:153
2851699405,21cyclictest22732-21ssh09:26:153
2851699401,16cyclictest0-21swapper/305:11:123
2851699401,16cyclictest0-21swapper/305:11:123
2851499405,21cyclictest13687-21ssh08:36:131
2851499401,6cyclictest12657-21switchtime06:36:281
28514994014,9cyclictest0-21swapper/108:41:491
28514994014,9cyclictest0-21swapper/108:41:491
2851499401,25cyclictest0-21swapper/109:01:131
2851699397,22cyclictest0-21swapper/308:31:153
2851699397,20cyclictest0-21swapper/305:16:123
2851699392,23cyclictest32202-21ls07:01:003
2851699391,25cyclictest0-21swapper/307:37:233
28515993919,4cyclictest7975-21sh09:07:222
28515993915,7cyclictest9130-21fschecks_time05:06:162
28515993911,15cyclictest7950irq/56-dwc_otg_08:31:032
28514993921,5cyclictest19067-21/usr/sbin/munin09:21:171
28514993918,6cyclictest327-21latency_hist07:40:591
2851499391,28cyclictest21341-21latency_hist05:41:001
8172380,3sleep30-21swapper/306:11:243
2851699387,22cyclictest0-21swapper/308:56:133
2851699385,27cyclictest7950irq/56-dwc_otg_07:21:093
2851699384,20cyclictest7950irq/56-dwc_otg_05:05:163
2851699381,18cyclictest0-21swapper/304:41:113
2851599389,20cyclictest25337-21runrttasks08:11:112
28515993826,10cyclictest7950irq/56-dwc_otg_08:56:132
28515993813,4cyclictest0-21swapper/207:16:122
2851599381,28cyclictest889-21ls08:20:592
2851599381,26cyclictest17471-21ls07:21:252
2851599381,17cyclictest14447-21rm08:36:222
2851499387,16cyclictest7950irq/56-dwc_otg_08:21:251
2851499384,7cyclictest7750irq/56-dwc_otg04:46:091
2851499382,21cyclictest19085-21ls10:01:001
28514993820,8cyclictest7950irq/56-dwc_otg_09:41:231
151032380,2sleep10-21swapper/109:16:221
75602370,1sleep10-21swapper/108:28:111
2851699378,6cyclictest39-21ksoftirqd/308:41:583
2851699378,6cyclictest39-21ksoftirqd/308:41:583
2851699378,5cyclictest0-21swapper/308:51:143
2851699377,15cyclictest7950irq/56-dwc_otg_07:51:273
2851699375,5cyclictest7950irq/56-dwc_otg_06:35:593
2851699375,2cyclictest0-21swapper/305:26:063
2851699374,7cyclictest2302-21ntp_kernel_pll_04:46:203
2851699372,6cyclictest39-21ksoftirqd/307:46:113
2851599373,3cyclictest7750irq/56-dwc_otg09:11:102
28515993717,5cyclictest16007-21gpg-connect-age05:25:592
28515993711,9cyclictest0-21swapper/209:26:232
28515993711,9cyclictest0-21swapper/209:26:232
2851599371,16cyclictest0-21swapper/207:11:112
2851499377,16cyclictest0-21swapper/107:31:131
2851499373,14cyclictest7950irq/56-dwc_otg_09:06:131
2851499371,27cyclictest17612-21timerwakeupswit07:21:261
2851499371,27cyclictest13285-21grep07:56:171
2851499370,24cyclictest0-21swapper/107:41:211
2851699369,15cyclictest381ktimersoftd/307:32:123
2851699366,5cyclictest37-21rcuc/307:26:123
2851699364,23cyclictest2667-21ssh09:41:133
2851699361,21cyclictest17480-21ls08:01:173
2851699361,21cyclictest17480-21ls08:01:173
2851699361,20cyclictest24896-21ntp_states06:51:233
28516993611,7cyclictest39-21ksoftirqd/309:18:193
2851599364,3cyclictest7750irq/56-dwc_otg05:16:122
2851599361,9cyclictest678-21ntp_kernel_pll_07:01:212
2851599361,9cyclictest678-21ntp_kernel_pll_07:01:212
28515993617,5cyclictest72672chrt08:27:392
28515993617,5cyclictest19367-21sh09:21:232
2851599361,3cyclictest0-21swapper/209:01:052
2851599361,27cyclictest22985-21ssh08:46:272
28515993612,5cyclictest7580-1kworker/u9:108:06:262
2851499369,12cyclictest7850irq/56-dwc_otg_08:06:161
2851499362,14cyclictest0-21swapper/105:01:101
2851699353,5cyclictest11693-21ssh06:36:143
28516993515,9cyclictest7950irq/56-dwc_otg_08:11:103
28516993511,5cyclictest0-21swapper/309:31:133
2851599353,13cyclictest7850irq/56-dwc_otg_08:26:002
2851599351,24cyclictest27904-21timerandwakeup05:56:272
2851499357,15cyclictest0-21swapper/106:31:141
2851499355,21cyclictest32393-21ssh07:01:111
2851499355,21cyclictest32393-21ssh07:01:111
28514993527,6cyclictest520-21cron06:30:591
2851499352,22cyclictest7850irq/56-dwc_otg_08:11:181
2851699347,24cyclictest0-21swapper/308:21:143
2851699343,9cyclictest0-21swapper/307:16:133
2851699343,22cyclictest7950irq/56-dwc_otg_05:56:173
2851699342,9cyclictest0-21swapper/306:24:473
2851699342,15cyclictest7750irq/56-dwc_otg09:21:253
2851599348,6cyclictest0-21swapper/207:37:142
2851599344,5cyclictest30105-21cron04:35:592
2851599344,5cyclictest30105-21cron04:35:592
2851599341,22cyclictest0-21swapper/204:56:122
2851599341,16cyclictest4119-21rm07:45:332
28515993410,5cyclictest0-21swapper/206:40:582
2851499348,11cyclictest7950irq/56-dwc_otg_05:51:141
2851499347,13cyclictest7950irq/56-dwc_otg_05:31:261
2851499346,8cyclictest0-21swapper/106:44:411
2851499344,13cyclictest7950irq/56-dwc_otg_04:36:121
2851499342,18cyclictest0-21swapper/106:23:441
28514993417,5cyclictest3508-21aten_rbpower_po04:51:121
28514993411,10cyclictest0-21swapper/108:16:501
2851699333,23cyclictest37-21rcuc/306:47:423
2851699331,3cyclictest7950irq/56-dwc_otg_05:41:093
28516993311,4cyclictest7950irq/56-dwc_otg_05:21:233
2851599339,13cyclictest7950irq/56-dwc_otg_08:51:142
2851599338,5cyclictest7950irq/56-dwc_otg_06:26:172
2851599334,14cyclictest7950irq/56-dwc_otg_06:51:172
28515993324,6cyclictest7054-21diskmemload07:26:102
2851599332,14cyclictest7950irq/56-dwc_otg_06:46:162
2851599331,20cyclictest32462-21wget06:11:122
2851499335,18cyclictest11040-21ssh09:51:131
2851499334,7cyclictest22306-21tune2fs08:46:171
28514993316,5cyclictest21607-21/usr/sbin/munin07:26:251
28514993315,6cyclictest8518-21grep07:11:171
2851499331,16cyclictest15984-21wc05:25:591
2851499331,16cyclictest10291-21apt-get09:11:111
28514993310,14cyclictest0-21swapper/107:20:371
109922330,3sleep10-21swapper/105:11:171
109922330,3sleep10-21swapper/105:11:171
2851699325,8cyclictest39-21ksoftirqd/306:10:583
2851699323,3cyclictest0-21swapper/305:46:083
2851699323,16cyclictest37-21rcuc/309:09:233
28516993210,5cyclictest0-21swapper/307:43:273
2851599327,7cyclictest0-21swapper/205:11:122
2851599327,7cyclictest0-21swapper/205:11:122
2851599325,13cyclictest7950irq/56-dwc_otg_04:41:122
2851599323,5cyclictest7750irq/56-dwc_otg06:01:182
2851599323,5cyclictest7750irq/56-dwc_otg06:01:182
28515993218,10cyclictest7950irq/56-dwc_otg_09:56:192
28515993215,5cyclictest3567-21/usr/sbin/munin06:21:142
2851599321,4cyclictest4084-21apt-get07:06:132
28515993211,6cyclictest1888-21tr06:16:132
2851599321,15cyclictest7850irq/56-dwc_otg_07:31:252
28515993210,5cyclictest31074-21/usr/sbin/munin09:36:142
2851499329,5cyclictest7850irq/56-dwc_otg_04:45:591
2851499327,5cyclictest7850irq/56-dwc_otg_05:20:581
2851499322,5cyclictest7750irq/56-dwc_otg06:56:121
2851499321,8cyclictest30151-21gpgv04:36:001
2851499321,8cyclictest30151-21gpgv04:36:001
2851499321,2cyclictest0-21swapper/106:01:111
2851499321,2cyclictest0-21swapper/106:01:111
28514993210,4cyclictest0-21swapper/106:48:491
2851699317,14cyclictest0-21swapper/304:36:053
2851599316,6cyclictest7950irq/56-dwc_otg_05:46:252
2851599316,15cyclictest0-21swapper/209:46:062
2851599315,13cyclictest7950irq/56-dwc_otg_05:01:242
2851599313,4cyclictest0-21swapper/206:56:152
2851599312,5cyclictest0-21swapper/208:01:202
2851599312,5cyclictest0-21swapper/208:01:202
2851599312,21cyclictest0-21swapper/209:52:062
2851599311,6cyclictest7850irq/56-dwc_otg_09:35:592
2851599311,5cyclictest31257-21/usr/sbin/munin04:36:232
28515993114,5cyclictest8893-21sh06:31:462
2851499317,5cyclictest7950irq/56-dwc_otg_04:56:231
2851499315,5cyclictest7750irq/56-dwc_otg05:06:131
2851499312,8cyclictest7750irq/56-dwc_otg06:11:121
2851499312,7cyclictest7750irq/56-dwc_otg06:06:131
28514993115,5cyclictest23771-21/usr/sbin/munin05:46:161
2851699307,20cyclictest0-21swapper/306:01:123
2851699307,20cyclictest0-21swapper/306:01:123
2851699307,20cyclictest0-21swapper/305:51:123
2851599307,20cyclictest0-21swapper/205:26:052
2851599304,5cyclictest7750irq/56-dwc_otg06:06:172
2851599302,5cyclictest7750irq/56-dwc_otg06:41:222
2851499304,7cyclictest21567-21/usr/sbin/munin05:41:201
28514993015,4cyclictest28106-21unixbench_multi05:56:291
2851699291,7cyclictest6382-21grep04:56:263
2851599296,11cyclictest7950irq/56-dwc_otg_05:51:212
2851599284,12cyclictest7950irq/56-dwc_otg_04:55:592
2851599281,18cyclictest0-21swapper/205:36:292
2851499281,5cyclictest16848-21ntp_states05:26:231
2851599272,13cyclictest7850irq/56-dwc_otg_05:31:202
2851499273,6cyclictest7950irq/56-dwc_otg_06:20:591
279632253,14sleep20-21swapper/204:27:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional