You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 21:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun Apr 12, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1491799986,77cyclictest0-21swapper/020:50:530
324032890,9sleep1221ktimersoftd/120:32:161
1491899836,36cyclictest7950irq/56-dwc_otg_17:00:551
1491899836,36cyclictest7950irq/56-dwc_otg_17:00:551
130022820,2sleep00-21swapper/020:48:170
215462810,2sleep20-21swapper/219:02:032
14917997421,16cyclictest0-21swapper/019:20:550
83592720,2sleep20-21swapper/220:03:512
14917997132,24cyclictest0-21swapper/018:40:550
12982710,2sleep20-21swapper/221:13:032
205672700,3sleep121-21rcuc/116:45:521
305322680,1sleep00-21swapper/017:11:070
198402670,1sleep30-21swapper/320:17:043
1491799661,6cyclictest19703-21sh20:56:060
14917996518,26cyclictest23660-21idleruntime-cro19:05:420
1491799651,33cyclictest0-21swapper/021:35:540
1491799647,23cyclictest0-21swapper/017:15:540
1491799621,57cyclictest25904-21cron18:30:420
144612612,52sleep30-21swapper/316:28:193
324862600,3sleep337-21rcuc/319:15:493
145312584,12sleep10-21swapper/116:29:141
1491799571,51cyclictest3704-21modprobe21:55:420
1491999567,28cyclictest0-21swapper/217:10:472
1491799567,28cyclictest0-21swapper/017:00:560
1491799567,28cyclictest0-21swapper/017:00:560
14917995531,11cyclictest908-21runrttasks19:08:560
14917995531,11cyclictest908-21runrttasks19:08:560
14917995524,16cyclictest0-21swapper/020:15:540
1491799552,28cyclictest30630-21/usr/sbin/munin18:36:100
14917995511,25cyclictest0-21swapper/016:35:480
1491799548,25cyclictest0-21swapper/020:00:590
14917995433,5cyclictest7750irq/56-dwc_otg19:41:110
14917995334,7cyclictest7750irq/56-dwc_otg18:15:420
14917995333,6cyclictest7750irq/56-dwc_otg19:31:070
14917995325,15cyclictest23064-21grep16:51:040
14917995323,12cyclictest29373-21sed21:46:090
14917995323,12cyclictest29373-21sed21:46:090
48952520,8sleep09-21ksoftirqd/020:38:390
48952520,8sleep09-21ksoftirqd/020:38:390
14917995229,6cyclictest7471-21wc18:46:020
14917995224,4cyclictest9616-21rm20:05:430
14917995224,4cyclictest9616-21rm20:05:430
14917995222,12cyclictest16076-21grep16:31:100
14917995221,4cyclictest7950irq/56-dwc_otg_20:11:100
1491899517,25cyclictest0-21swapper/117:50:571
1491899511,7cyclictest7950irq/56-dwc_otg_19:10:531
1491799516,24cyclictest0-21swapper/020:35:400
1491799514,14cyclictest0-21swapper/019:25:540
14917995131,7cyclictest7750irq/56-dwc_otg18:50:580
14917995129,5cyclictest11089-21latency_hist21:25:430
14917995125,11cyclictest0-21swapper/019:36:100
14917995123,13cyclictest9208-21sed17:41:110
14917995121,16cyclictest0-21swapper/019:50:420
1491799511,26cyclictest24978-21timerandwakeup21:41:090
1491799511,26cyclictest24978-21timerandwakeup21:41:090
14918995022,5cyclictest1394150wget20:10:561
1491799509,22cyclictest0-21swapper/017:20:570
14917995023,15cyclictest22578-21fschecks_time20:21:000
14917995021,15cyclictest0-21swapper/017:35:420
14917995021,15cyclictest0-21swapper/017:35:420
1491899497,18cyclictest0-21swapper/118:50:551
14917994935,5cyclictest0-21swapper/019:10:530
14917994932,7cyclictest7850irq/56-dwc_otg_21:26:070
14917994928,6cyclictest19556-21latency_hist21:35:430
1491799492,25cyclictest13582-21aten_rbpower_en17:55:560
14917994921,16cyclictest1313-21grep19:16:100
14917994919,14cyclictest6067-21sh21:19:120
1491899487,24cyclictest0-21swapper/117:10:471
14918994823,5cyclictest6057-21idleruntime-cro20:40:431
14918994823,5cyclictest6057-21idleruntime-cro20:40:431
14917994824,4cyclictest30070-21grep19:51:000
14917994818,15cyclictest12705-21cat17:51:110
1491799481,32cyclictest0-21swapper/021:00:530
1491799481,29cyclictest0-21swapper/020:26:050
1491799481,27cyclictest31441-21seq21:10:560
1491799481,14cyclictest28033-21ssh21:06:050
14919994717,18cyclictest0-21swapper/219:45:422
14919994713,4cyclictest7950irq/56-dwc_otg_17:21:042
1491799471,16cyclictest0-21swapper/018:31:310
14919994613,20cyclictest7850irq/56-dwc_otg_18:50:552
1491899467,25cyclictest0-21swapper/117:05:561
14917994627,5cyclictest6437-21aten_rbpower_en20:40:560
14917994625,5cyclictest19929-21grep16:41:100
14917994623,6cyclictest7750irq/56-dwc_otg19:56:060
1491799461,25cyclictest0-21swapper/018:06:050
1491799461,22cyclictest7891-21grep22:00:420
174612440,5sleep11491899cyclictest21:32:151
1491999447,25cyclictest0-21swapper/217:05:562
1491999446,27cyclictest15573-21chrt21:30:552
14917994422,10cyclictest11482-21munin-run17:50:420
14917994419,11cyclictest0-21swapper/018:21:020
14917994412,4cyclictest908-21runrttasks16:49:250
145032443,35sleep00-21swapper/016:28:520
14920994337,4cyclictest7284-21sh21:20:583
1491999431,5cyclictest27230-21apt-get21:05:532
14918994312,22cyclictest0-21swapper/117:20:561
14917994326,4cyclictest0-21swapper/018:55:550
14917994317,14cyclictest0-21swapper/016:56:050
60542420,2sleep10-21swapper/118:45:031
311942420,4sleep31492099cyclictest21:10:433
1491999421,26cyclictest0-21swapper/218:45:472
1491899427,6cyclictest7950irq/56-dwc_otg_18:46:071
1491899424,3cyclictest14779-21ssh20:50:531
14917994223,9cyclictest6343-21aten_rbpower_en17:35:550
14917994213,14cyclictest28035-21/usr/sbin/munin17:06:000
225912411,34sleep10-21swapper/120:21:021
14917994124,4cyclictest0-21swapper/018:01:010
1491999407,23cyclictest0-21swapper/218:25:572
1491999407,18cyclictest7950irq/56-dwc_otg_20:25:542
1491899408,5cyclictest7750irq/56-dwc_otg18:56:021
1491899406,14cyclictest2760-21vmstat19:56:121
1491899402,4cyclictest0-21swapper/119:50:191
14918994018,4cyclictest15251-21latency_hist21:30:421
14918994016,6cyclictest19142-21/usr/sbin/munin18:10:571
14917994013,13cyclictest21313-21cat18:16:040
14917994013,13cyclictest21313-21cat18:16:040
126852400,3sleep337-21rcuc/319:30:423
1491999391,4cyclictest7850irq/56-dwc_otg_21:45:592
1491999391,4cyclictest7850irq/56-dwc_otg_21:45:592
1491999391,25cyclictest0-21swapper/217:15:542
1491899395,22cyclictest18048-21ssh20:15:551
14918993913,5cyclictest25929-21diskmemload19:35:361
14917993922,11cyclictest0-21swapper/017:28:580
109742390,4sleep310953-21ssh20:06:093
109742390,4sleep310953-21ssh20:06:093
44282380,4sleep2301ktimersoftd/219:20:552
23082380,1sleep20-21swapper/221:53:132
157042380,2sleep10-21swapper/116:31:071
15342381,4sleep30-21swapper/321:13:293
14919993816,13cyclictest16849-21apt16:35:522
1491899389,4cyclictest0-21swapper/120:26:061
14918993816,4cyclictest2446050wget18:25:581
14918993813,5cyclictest25929-21diskmemload21:41:021
14918993813,5cyclictest25929-21diskmemload21:41:021
1491899381,17cyclictest585-21if_wlan019:16:011
14918993811,17cyclictest0-21swapper/121:47:401
14918993811,17cyclictest0-21swapper/121:47:401
14920993714,4cyclictest22493-21aten_rbpower_vo16:50:553
1491899378,4cyclictest0-21swapper/121:08:181
1491899376,15cyclictest0-21swapper/117:15:541
1491899373,8cyclictest0-21swapper/121:11:001
14918993724,6cyclictest221ktimersoftd/117:45:531
1492099362,20cyclictest0-21swapper/321:01:013
1491999368,7cyclictest18477-21unixbench_multi19:36:102
14919993619,5cyclictest22576-21sh20:21:002
14919993619,5cyclictest15833-21fschecks_time18:56:002
14919993616,5cyclictest18034-21sh20:15:542
1491999361,4cyclictest0-21swapper/218:36:012
14919993613,5cyclictest11979-21fschecks_count17:51:012
1491999361,22cyclictest0-21swapper/218:20:542
1491899369,17cyclictest0-21swapper/119:23:431
1491899368,4cyclictest0-21swapper/118:37:311
1491899366,4cyclictest0-21swapper/119:51:031
1491899362,9cyclictest7424-21grep21:21:001
14918993621,4cyclictest18426-21sh19:36:101
14918993615,6cyclictest6030-21cron17:35:411
14918993615,6cyclictest6030-21cron17:35:411
1491899361,3cyclictest0-21swapper/120:47:571
14918993610,4cyclictest0-21swapper/121:04:411
14918993610,20cyclictest14914-21cyclictest21:37:251
115762360,1sleep30-21swapper/321:25:563
1492099358,5cyclictest7850irq/56-dwc_otg_19:55:573
1492099351,2cyclictest0-21swapper/318:35:543
1491999359,7cyclictest16700-21ssh20:52:162
1491999357,5cyclictest8346-21ls21:21:132
1491999356,15cyclictest0-21swapper/219:05:552
1491999356,15cyclictest0-21swapper/219:05:552
1491999353,5cyclictest7053-21latency_hist21:20:432
14919993515,13cyclictest120922chrt20:08:052
14919993515,13cyclictest120922chrt20:08:052
14919993514,4cyclictest25929-21diskmemload20:41:472
1491999351,20cyclictest0-21swapper/219:57:172
1491899357,4cyclictest7750irq/56-dwc_otg17:41:011
1491899357,4cyclictest0-21swapper/119:02:261
1491899356,5cyclictest24170-21wc18:25:421
14918993519,4cyclictest6814-21unixbench_multi20:01:101
14918993515,9cyclictest26352-21ssh19:08:001
14918993515,9cyclictest26352-21ssh19:08:001
14918993513,5cyclictest18443-21basename16:40:421
1491899351,13cyclictest7950irq/56-dwc_otg_20:10:421
1491899351,13cyclictest7950irq/56-dwc_otg_20:10:421
1492099349,7cyclictest7750irq/56-dwc_otg21:36:043
1492099346,16cyclictest7950irq/56-dwc_otg_20:00:573
14920993417,4cyclictest310-21if_wlan021:51:013
14920993415,4cyclictest20600-21basename16:45:553
1492099341,15cyclictest7750irq/56-dwc_otg20:11:003
14920993411,5cyclictest27987-21sh20:26:463
14920993410,4cyclictest0-21swapper/320:21:023
1491999346,16cyclictest0-21swapper/221:40:552
1491999346,16cyclictest0-21swapper/221:40:552
14919993422,7cyclictest28282-21seq19:10:552
14919993411,12cyclictest0-21swapper/219:25:592
1491899347,4cyclictest7850irq/56-dwc_otg_16:40:431
1491899345,4cyclictest7750irq/56-dwc_otg18:31:031
1491899345,4cyclictest0-21swapper/120:58:411
1491899341,21cyclictest12792-21ssh19:30:431
38632330,3sleep231-21ksoftirqd/220:36:482
38632330,3sleep231-21ksoftirqd/220:36:482
268382330,23sleep27950irq/56-dwc_otg_17:01:072
268382330,23sleep27950irq/56-dwc_otg_17:01:072
1492099334,4cyclictest7750irq/56-dwc_otg21:16:043
1492099332,6cyclictest7850irq/56-dwc_otg_16:40:483
14920993326,4cyclictest14586-21latency_hist20:50:433
1492099332,15cyclictest7850irq/56-dwc_otg_19:55:423
14920993316,6cyclictest3783-21grep17:26:103
14920993313,13cyclictest15122-21fschecks_time20:50:573
1491999336,5cyclictest7850irq/56-dwc_otg_18:30:482
1491999335,10cyclictest25929-21diskmemload19:53:012
1491999331,8cyclictest0-21swapper/217:56:022
1491999331,3cyclictest0-21swapper/221:28:002
1491999331,20cyclictest0-21swapper/221:38:152
1491999331,18cyclictest13299-21fschecks_time19:30:592
1491999331,15cyclictest7950irq/56-dwc_otg_16:48:322
1491899337,4cyclictest0-21swapper/120:41:031
1491899335,6cyclictest7950irq/56-dwc_otg_17:56:031
1491899335,5cyclictest7950irq/56-dwc_otg_21:16:111
1491899331,5cyclictest482950ssh21:56:081
14918993313,11cyclictest24927-21proc_pri16:56:081
1492099329,6cyclictest0-21swapper/321:50:433
1492099329,6cyclictest0-21swapper/321:50:433
1492099329,11cyclictest0-21swapper/320:39:403
1492099329,11cyclictest0-21swapper/320:39:403
1492099326,6cyclictest37-21rcuc/319:21:023
1492099324,5cyclictest7750irq/56-dwc_otg21:30:583
1492099323,8cyclictest7750irq/56-dwc_otg17:15:543
14920993226,4cyclictest17991-21ls18:06:083
14920993211,4cyclictest0-21swapper/318:42:433
1491999329,5cyclictest6562-21tune2fs17:35:582
1491999324,5cyclictest7750irq/56-dwc_otg20:56:132
1491999322,17cyclictest12859-21kworker/u8:019:46:032
14919993217,4cyclictest11478-21latency_hist17:50:432
1491999321,21cyclictest10454-21ssh20:45:482
1491999321,16cyclictest7950irq/56-dwc_otg_18:43:442
1491999321,15cyclictest7950irq/56-dwc_otg_21:56:102
1491899328,11cyclictest7850irq/56-dwc_otg_21:51:051
1491899327,5cyclictest7950irq/56-dwc_otg_17:35:581
14918993211,10cyclictest17254-21cpu18:05:571
1492099319,10cyclictest0-21swapper/319:11:323
1492099316,6cyclictest7850irq/56-dwc_otg_18:21:013
1492099313,15cyclictest25984-21/usr/sbin/munin17:01:073
1492099313,15cyclictest25984-21/usr/sbin/munin17:01:073
1492099312,9cyclictest0-21swapper/318:00:553
1492099312,4cyclictest0-21swapper/318:55:543
1492099312,13cyclictest7850irq/56-dwc_otg_19:05:553
1492099312,13cyclictest7850irq/56-dwc_otg_19:05:553
14920993110,3cyclictest0-21swapper/319:40:553
1491999317,10cyclictest0-21swapper/216:40:432
1491999311,21cyclictest0-21swapper/220:13:262
14919993111,13cyclictest40932chrt17:29:002
1491899315,10cyclictest7950irq/56-dwc_otg_16:50:551
14918993111,6cyclictest22616-21/usr/sbin/munin19:41:101
128922317,11sleep20-21swapper/216:25:432
1492099309,6cyclictest15116-21awk18:55:423
1492099308,5cyclictest7750irq/56-dwc_otg20:31:023
1492099306,7cyclictest7950irq/56-dwc_otg_17:06:113
1492099305,12cyclictest7950irq/56-dwc_otg_17:41:063
1492099303,4cyclictest7850irq/56-dwc_otg_20:41:003
14920993024,4cyclictest25924-21apt-get18:30:423
14920993023,5cyclictest26494-21aten_rbpower_vo18:30:553
1492099302,25cyclictest0-21swapper/318:48:533
1492099301,7cyclictest0-21swapper/319:50:013
1492099301,15cyclictest0-21swapper/321:56:013
1491999307,13cyclictest23648-21kworker/2:016:56:072
1491999306,15cyclictest0-21swapper/217:30:562
1491999306,15cyclictest0-21swapper/217:30:562
1491999304,5cyclictest16104-21head16:31:102
1491999301,18cyclictest22996-21ssh21:00:452
1491999301,17cyclictest0-21swapper/218:16:062
1491999301,17cyclictest0-21swapper/218:16:062
1491999301,11cyclictest0-21swapper/218:05:572
1491899306,4cyclictest0-21swapper/117:26:091
1492099293,15cyclictest23766-21latency_hist19:05:433
1492099292,7cyclictest27721-21munin-run21:45:423
1492099292,7cyclictest27721-21munin-run21:45:423
1492099292,5cyclictest7850irq/56-dwc_otg_17:46:043
1492099292,18cyclictest0-21swapper/317:22:123
1492099291,25cyclictest16552-21apt-key16:35:433
14920992910,5cyclictest917-21snmpd19:36:013
14920992910,4cyclictest0-21swapper/319:30:533
1491999292,16cyclictest0-21swapper/220:32:372
1491999291,5cyclictest872-21ssh19:16:052
14919992915,4cyclictest19053-21cpu18:10:562
14918992914,5cyclictest21399-21netstat18:16:041
14918992914,5cyclictest21399-21netstat18:16:041
1492099287,4cyclictest7750irq/56-dwc_otg17:56:063
1492099284,11cyclictest7850irq/56-dwc_otg_21:00:423
1492099281,14cyclictest7950irq/56-dwc_otg_17:10:553
1491999285,5cyclictest9266-21grep17:41:112
14919992811,10cyclictest23105-21/usr/sbin/munin16:51:062
1492099277,9cyclictest7950irq/56-dwc_otg_17:36:083
1492099273,14cyclictest0-21swapper/318:15:583
1492099273,14cyclictest0-21swapper/318:15:583
1492099272,12cyclictest7850irq/56-dwc_otg_16:35:553
14920992720,5cyclictest24249-21aten_rbpower_en16:55:573
1492099271,18cyclictest19947-21timerwakeupswit18:11:093
1492099262,4cyclictest0-21swapper/317:50:543
1491999262,14cyclictest0-21swapper/218:01:092
1491899261,4cyclictest16343-21timerwakeupswit18:01:091
1492099231,10cyclictest7750irq/56-dwc_otg17:31:023
1492099231,10cyclictest7750irq/56-dwc_otg17:31:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional