You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue Sep 16, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
922321620,3sleep196599cyclictest23:55:371
922321620,3sleep196599cyclictest23:55:371
96499891,5cyclictest11536-21latency_hist22:35:020
96499873,76cyclictest0-21swapper/000:20:130
96499841,5cyclictest30962-21ssh21:35:120
96499821,64cyclictest2033-21date23:05:020
96499811,5cyclictest9953-21cron21:05:010
96499811,5cyclictest7153-21grep23:10:320
96499798,65cyclictest16777-21unixbench_multi22:40:310
96499791,5cyclictest25269-21df23:35:180
96499782,6cyclictest15616-21aten_rbpower_en19:50:150
964997826,43cyclictest7950irq/56-dwc_otg_20:15:210
96499781,5cyclictest31031-21unixbench_multi00:25:300
96499781,5cyclictest31031-21unixbench_multi00:25:300
96499781,5cyclictest26521-21unixbench_singl22:10:310
300712770,3sleep10-21swapper/122:15:301
96499767,58cyclictest1343-21ls20:40:170
96499762,7cyclictest6530-21tr19:25:140
270982750,2sleep30-21swapper/320:20:233
96499742,6cyclictest13257-21ssh21:10:300
96499742,4cyclictest0-21swapper/023:30:100
96499741,6cyclictest12141-21grep22:35:210
96499741,54cyclictest18130-21ssh21:17:490
79942740,5sleep296699cyclictest22:30:162
96499739,53cyclictest22586-21ssh21:24:180
96499739,49cyclictest21528-21aten_rbpower_en22:05:140
96499732,5cyclictest0-21swapper/021:54:170
96499732,3cyclictest0-21swapper/021:46:070
96499731,60cyclictest11774-21diskmemload23:21:100
96499731,60cyclictest11774-21diskmemload23:21:100
96499731,60cyclictest0-21swapper/019:14:370
96499728,61cyclictest12217-21if_eth000:00:220
96499721,55cyclictest30529-21ssh23:41:150
96499715,56cyclictest3346-21ssh23:49:150
96499711,56cyclictest28810-21ssh22:56:320
964997111,40cyclictest10193-21munin-node21:05:110
164072710,2sleep20-21swapper/221:58:122
964997017,33cyclictest0-21swapper/022:15:280
96499701,6cyclictest26731-21idleruntime-cro21:30:010
964997011,46cyclictest3040-21awk23:05:290
964997010,46cyclictest3157-21ssh00:33:540
137272700,1sleep00-21swapper/021:55:040
964996921,40cyclictest0-21swapper/023:25:220
964996914,34cyclictest10742-21rm23:15:290
964996912,42cyclictest248432sleep000:17:100
96499682,5cyclictest0-21swapper/019:45:010
96499682,5cyclictest0-21swapper/019:45:010
96499682,4cyclictest0-21swapper/020:10:020
96499681,6cyclictest140-21systemd-journal23:55:020
96499681,6cyclictest140-21systemd-journal23:55:020
96499681,6cyclictest0-21swapper/000:08:400
96499679,48cyclictest15135-21taskset19:47:540
96499679,39cyclictest0-21swapper/022:48:180
96499679,39cyclictest0-21swapper/022:48:180
96499676,57cyclictest140-21systemd-journal22:30:010
96499675,54cyclictest710-21chrt20:37:350
96499675,48cyclictest23210-21apt-get22:50:160
96499671,7cyclictest0-21swapper/022:21:500
964996710,42cyclictest17310-21apt-get22:00:070
65382670,3sleep15909-21/usr/sbin/munin23:10:241
96499661,4cyclictest0-21swapper/023:51:550
965996515,46cyclictest4311-21aten_rbpower_po22:25:161
96499659,42cyclictest8165-21basename21:00:010
96499651,7cyclictest0-21swapper/000:14:350
96499651,44cyclictest0-21swapper/021:42:510
96599647,4cyclictest0-21swapper/123:08:101
965996424,29cyclictest8185-21latency_hist21:00:011
96499646,44cyclictest0-21swapper/019:31:150
96499642,5cyclictest0-21swapper/020:48:340
96499642,4cyclictest0-21swapper/000:36:580
96499641,6cyclictest31671-21latency_hist20:35:020
96499641,6cyclictest31671-21latency_hist20:35:020
964996415,4cyclictest10175-21aten_rbpower_cu19:35:150
96599633,6cyclictest0-21swapper/122:05:141
96599631,5cyclictest4548-21iwlist23:50:201
96499631,8cyclictest0-21swapper/019:16:560
96499631,40cyclictest0-21swapper/021:31:360
96599624,12cyclictest7950irq/56-dwc_otg_19:50:231
96599622,14cyclictest0-21swapper/121:13:461
96599611,9cyclictest25770-21ntpq23:35:251
96599609,4cyclictest7950irq/56-dwc_otg_22:31:171
96599605,5cyclictest7206-21ntp_kernel_pll_19:25:251
96599603,8cyclictest0-21swapper/121:25:131
965996018,37cyclictest18809-21sh00:10:011
965996011,4cyclictest0-21swapper/123:31:301
96499607,40cyclictest6114-21seq19:22:570
96499601,48cyclictest0-21swapper/019:55:520
964996010,40cyclictest29690-21sshd20:29:150
967995910,22cyclictest0-21swapper/300:05:063
965995915,5cyclictest32703-21apt-get00:30:101
96499592,47cyclictest0-21swapper/020:21:150
45582590,2sleep20-21swapper/220:50:012
96599581,4cyclictest7950irq/56-dwc_otg_21:05:131
96499581,2cyclictest0-21swapper/020:50:280
96599574,4cyclictest7850irq/56-dwc_otg22:40:231
96599572,5cyclictest12523-21ntp_states00:00:271
96599572,4cyclictest7850irq/56-dwc_otg21:40:301
96599571,5cyclictest0-21swapper/122:55:011
964995713,28cyclictest0-21swapper/020:12:570
96799568,30cyclictest0-21swapper/321:50:153
96699565,11cyclictest0-21swapper/223:55:122
96699565,11cyclictest0-21swapper/223:55:122
96599568,4cyclictest7950irq/56-dwc_otg_22:55:451
96599561,9cyclictest0-21swapper/100:20:181
96599561,6cyclictest1843-21ntp_kernel_err20:40:241
96599561,5cyclictest8050irq/56-dwc_otg_22:24:201
96599561,3cyclictest8050irq/56-dwc_otg_20:15:261
125112560,2sleep30-21swapper/319:40:233
125112560,2sleep30-21swapper/319:40:233
96599558,13cyclictest0-21swapper/119:35:151
96599555,6cyclictest8050irq/56-dwc_otg_00:20:021
96599552,3cyclictest0-21swapper/121:35:191
96499551,41cyclictest0-21swapper/020:01:470
96699546,15cyclictest7850irq/56-dwc_otg00:05:062
96699542,11cyclictest0-21swapper/200:00:152
96599542,4cyclictest0-21swapper/121:20:251
96599541,7cyclictest16815-21ssh23:24:251
96599541,7cyclictest16815-21ssh23:24:251
96599541,4cyclictest22774-21/usr/sbin/munin20:10:271
96599541,3cyclictest0-21swapper/121:55:211
96599541,3cyclictest0-21swapper/120:50:271
965995413,4cyclictest25397-21cpu22:10:171
965995412,4cyclictest18195-21/usr/sbin/munin19:55:261
96599541,10cyclictest8050irq/56-dwc_otg_23:46:301
965995410,6cyclictest29216-21ssh23:40:211
96699532,27cyclictest0-21swapper/200:10:172
96599536,7cyclictest8050irq/56-dwc_otg_22:46:261
96599536,7cyclictest8050irq/56-dwc_otg_22:46:261
96599531,5cyclictest6559-21taskset00:39:321
96599523,8cyclictest7850irq/56-dwc_otg23:25:171
96599522,5cyclictest0-21swapper/123:16:071
96599522,4cyclictest0-21swapper/121:30:081
96599522,3cyclictest0-21swapper/120:20:261
96599521,6cyclictest19788-21ssh22:02:081
96599511,3cyclictest0-21swapper/121:50:041
96799507,14cyclictest0-21swapper/322:00:083
96599502,8cyclictest0-21swapper/119:20:131
96599502,4cyclictest7850irq/56-dwc_otg21:15:251
96599502,2cyclictest0-21swapper/100:14:051
96599501,6cyclictest739-21chrt20:37:561
96599501,5cyclictest12390-21missed_timers22:35:251
96599497,26cyclictest0-21swapper/120:05:181
96599493,8cyclictest0-21swapper/120:46:161
96599492,4cyclictest7850irq/56-dwc_otg21:45:581
965994914,4cyclictest9066-21/usr/sbin/munin19:30:251
96599491,3cyclictest7850irq/56-dwc_otg00:25:351
96599491,3cyclictest7850irq/56-dwc_otg00:25:351
96599482,9cyclictest8050irq/56-dwc_otg_20:26:311
96599481,4cyclictest1188-21ssh23:03:401
965994813,4cyclictest15314-21latency_hist19:50:011
96799477,23cyclictest0-21swapper/322:25:153
965994617,5cyclictest2968-21aten_rbpower_en19:15:181
96799447,23cyclictest0-21swapper/322:45:153
96799447,23cyclictest0-21swapper/322:45:153
96799446,17cyclictest39-21ksoftirqd/319:25:163
96799441,11cyclictest0-21swapper/300:10:173
96599442,3cyclictest7850irq/56-dwc_otg21:00:311
4942443,34sleep00-21swapper/019:07:310
152192440,2sleep30-21swapper/300:05:013
967994317,15cyclictest381ktimersoftd/321:10:133
96599437,24cyclictest0-21swapper/119:40:151
96599437,24cyclictest0-21swapper/119:40:151
96799427,22cyclictest0-21swapper/321:20:073
966994221,5cyclictest2179-21ls21:40:022
96799412,6cyclictest7850irq/56-dwc_otg23:55:143
96799412,6cyclictest7850irq/56-dwc_otg23:55:143
96799412,4cyclictest0-21swapper/323:10:153
967994116,5cyclictest22882-21basename00:15:163
96699417,23cyclictest0-21swapper/220:50:192
96699415,6cyclictest32680-21rm23:45:132
966994120,4cyclictest19405-21sh22:45:152
966994120,4cyclictest19405-21sh22:45:152
966994116,5cyclictest8050irq/56-dwc_otg_21:50:132
96799404,4cyclictest0-21swapper/323:45:133
967994015,6cyclictest39-21ksoftirqd/322:35:013
96699407,25cyclictest0-21swapper/221:20:072
96699407,24cyclictest0-21swapper/220:30:122
96699407,24cyclictest0-21swapper/220:30:122
96699406,22cyclictest0-21swapper/220:15:152
96699402,4cyclictest7850irq/56-dwc_otg23:35:162
49182400,2sleep30-21swapper/321:43:113
158062400,1sleep30-21swapper/321:56:563
96799396,20cyclictest39-21ksoftirqd/323:05:023
96799395,22cyclictest28893-21ssh23:40:163
96799391,23cyclictest29795-21ls20:30:023
967993910,4cyclictest7850irq/56-dwc_otg21:35:293
96699399,13cyclictest31-21ksoftirqd/221:05:012
966993918,14cyclictest31-21ksoftirqd/222:35:192
966993913,13cyclictest0-21swapper/221:10:312
96699391,14cyclictest8050irq/56-dwc_otg_23:10:152
96799388,5cyclictest7950irq/56-dwc_otg_21:15:233
96799382,3cyclictest0-21swapper/323:25:193
96799382,30cyclictest0-21swapper/322:20:123
967993819,10cyclictest1111-21runrttasks21:27:553
96699387,22cyclictest31-21ksoftirqd/200:30:092
96699386,22cyclictest0-21swapper/223:15:152
966993814,6cyclictest7950irq/56-dwc_otg_22:05:272
96699381,24cyclictest2744-21latency_hist20:45:012
96599381,25cyclictest1524-21latency19:10:211
96799372,5cyclictest8050irq/56-dwc_otg_20:05:183
967993717,6cyclictest7850irq/56-dwc_otg22:37:453
96799371,28cyclictest27441-21ls00:20:303
96799371,24cyclictest8836-21latency19:30:213
966993721,4cyclictest3169-21sh23:05:302
966993719,4cyclictest26904-21latency_hist21:30:022
966993715,7cyclictest29351-21fschecks_count22:15:202
96799367,12cyclictest8050irq/56-dwc_otg_00:25:233
96799367,12cyclictest8050irq/56-dwc_otg_00:25:233
96799366,10cyclictest39-21ksoftirqd/319:50:163
96799361,9cyclictest37-21rcuc/323:54:363
967993618,4cyclictest39-21ksoftirqd/321:05:163
967993613,4cyclictest37-21rcuc/320:40:293
967993610,9cyclictest8050irq/56-dwc_otg_23:05:233
96699367,19cyclictest0-21swapper/219:15:072
966993617,4cyclictest1111-21runrttasks00:19:442
966993614,5cyclictest29-21rcuc/223:30:012
96699361,24cyclictest5200-21ls22:25:282
96799352,14cyclictest381ktimersoftd/323:30:123
967993518,4cyclictest37-21rcuc/321:46:233
967993516,6cyclictest7950irq/56-dwc_otg_20:15:183
96799351,16cyclictest27876-21ssh21:30:283
967993511,10cyclictest39-21ksoftirqd/322:40:313
96699359,12cyclictest31-21ksoftirqd/223:32:262
96699358,4cyclictest7574-21latency_hist23:55:012
96699358,17cyclictest7950irq/56-dwc_otg_23:05:012
96699353,13cyclictest8050irq/56-dwc_otg_20:55:282
96699352,30cyclictest0-21swapper/222:55:122
966993520,7cyclictest7950irq/56-dwc_otg_21:47:082
96799347,21cyclictest0-21swapper/319:15:073
96799342,3cyclictest7950irq/56-dwc_otg_23:35:153
967993417,5cyclictest381ktimersoftd/322:17:423
96799341,5cyclictest39-21ksoftirqd/322:05:243
96699349,17cyclictest7950irq/56-dwc_otg_19:50:152
966993416,9cyclictest301ktimersoftd/200:23:052
966993413,9cyclictest29-21rcuc/219:22:232
96699341,22cyclictest11399-21kworker/u8:222:40:242
96699341,11cyclictest8050irq/56-dwc_otg_19:50:012
96599341,6cyclictest0-21swapper/120:32:221
96599341,6cyclictest0-21swapper/120:32:221
96599341,6cyclictest0-21swapper/120:04:351
96799337,7cyclictest39-21ksoftirqd/323:18:423
96799337,16cyclictest39-21ksoftirqd/322:13:133
96799337,14cyclictest0-21swapper/320:45:163
96799336,11cyclictest7950irq/56-dwc_otg_00:39:213
96799333,8cyclictest7950irq/56-dwc_otg_22:51:223
967993316,5cyclictest39-21ksoftirqd/322:59:283
96799331,4cyclictest10001-21latency_hist21:05:013
967993314,4cyclictest37-21rcuc/320:59:493
96799331,22cyclictest0-21swapper/320:30:123
96799331,22cyclictest0-21swapper/320:30:123
96799331,21cyclictest0-21swapper/300:30:093
96799331,19cyclictest18927-21latency_hist20:00:013
96699336,15cyclictest0-21swapper/219:25:162
96699335,20cyclictest0-21swapper/223:20:152
96699335,20cyclictest0-21swapper/223:20:152
96699333,15cyclictest0-21swapper/221:30:162
96699331,6cyclictest30214-21fschecks_time00:25:192
96699331,6cyclictest30214-21fschecks_time00:25:192
96699331,6cyclictest26516-21grep20:20:152
966993316,5cyclictest31-21ksoftirqd/222:10:262
96699331,24cyclictest3323-21ssh22:23:522
96699331,22cyclictest4155-21uniq00:35:202
96799328,13cyclictest39-21ksoftirqd/319:20:023
967993219,5cyclictest13918-21cut23:20:203
967993219,5cyclictest13918-21cut23:20:203
96699328,11cyclictest0-21swapper/222:50:302
96699327,14cyclictest0-21swapper/221:18:282
96699325,9cyclictest8050irq/56-dwc_otg_21:05:242
96699323,16cyclictest29-21rcuc/223:43:322
96699321,5cyclictest11301-21unixbench_singl19:35:292
96699321,21cyclictest914-21latency_hist20:40:012
96699321,14cyclictest0-21swapper/219:30:282
966993211,11cyclictest29-21rcuc/219:42:092
966993211,11cyclictest29-21rcuc/219:42:092
96799316,15cyclictest0-21swapper/320:10:183
96699315,6cyclictest31-21ksoftirqd/221:40:332
966993123,5cyclictest301ktimersoftd/220:10:182
966993114,8cyclictest11774-21diskmemload22:01:312
96699311,23cyclictest21493-21latency20:05:242
96699311,21cyclictest2606-21latency_hist19:15:012
967993012,4cyclictest1111-21runrttasks20:52:193
96699306,13cyclictest7850irq/56-dwc_otg20:26:442
96799295,16cyclictest7950irq/56-dwc_otg_19:36:403
96799287,6cyclictest37-21rcuc/320:04:223
96799286,6cyclictest8050irq/56-dwc_otg_19:45:023
96799281,25cyclictest32300-21latency20:35:233
96699287,13cyclictest20259-21unixbench_multi20:00:302
3282289,11sleep10-21swapper/119:05:331
96699272,12cyclictest8050irq/56-dwc_otg_19:58:512
96799264,7cyclictest1111-21runrttasks19:12:283
322812263,12sleep20-21swapper/219:05:262
3352253,14sleep30-21swapper/319:05:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional