You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 06:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Mon May 25, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1409521830,5sleep0325199cyclictest00:25:040
32519910643,16cyclictest0-21swapper/022:25:150
32519910144,30cyclictest0-21swapper/023:55:140
3251998618,46cyclictest8741-21grep19:25:170
3251998616,58cyclictest17583-21proc_pri23:10:270
3251998516,54cyclictest3647-21if_eth021:35:220
118752850,2sleep10-21swapper/121:00:301
3251998428,34cyclictest19629-21apt-get19:55:150
3251998421,47cyclictest15590-21ntpq21:10:290
3251998419,43cyclictest0-21swapper/023:25:030
3251998219,47cyclictest14118-21ls23:45:250
3251998118,41cyclictest20902-21tail21:55:310
3251998117,51cyclictest23211-21ssh22:38:310
3251998019,40cyclictest4673-21chrt22:15:290
325199797,59cyclictest0-21swapper/021:00:170
3251997920,40cyclictest31163-21grep00:05:350
3251997916,46cyclictest7939-21ntpq20:50:260
3251997817,48cyclictest31949-21ssh23:27:550
3251997817,48cyclictest31949-21ssh23:27:550
3251997719,45cyclictest30562-21/usr/sbin/munin20:25:260
3251997719,41cyclictest7972-21sh00:17:020
3251997716,52cyclictest15021-21ssh23:07:320
3251997515,49cyclictest0-21swapper/000:11:190
3251997512,41cyclictest0-21swapper/022:14:190
3251997415,50cyclictest26146-21runrttasks00:39:500
3251997414,46cyclictest0-21swapper/022:06:570
182932740,1sleep10-21swapper/122:32:041
3251997318,46cyclictest1-21systemd00:31:190
3251997315,45cyclictest15707-21taskset19:44:160
3251997313,45cyclictest0-21swapper/023:51:280
3251997311,20cyclictest0-21swapper/021:10:040
3251997218,40cyclictest12492-21ssh00:22:440
3251997217,39cyclictest24191-21ssh23:19:130
3251997217,38cyclictest1672-21ls23:30:260
3251997215,43cyclictest28216-21ssh21:25:280
3251997214,43cyclictest19762-21ssh21:15:280
3251997214,43cyclictest19762-21ssh21:15:280
3251997214,40cyclictest22628-21taskset20:00:380
3251997213,51cyclictest11817-21chrt19:30:400
3251997212,44cyclictest0-21swapper/022:01:120
198272720,2sleep00-21swapper/022:34:530
3251997123,24cyclictest5817-21/usr/sbin/munin23:35:270
3251997122,40cyclictest10466-21ssh23:01:470
3251997121,31cyclictest26181-21ssh22:41:240
3251997119,40cyclictest18017-21ssh21:52:350
3251997119,40cyclictest18017-21ssh21:52:350
3251997116,40cyclictest7898-21ssh22:20:160
3251997116,40cyclictest7898-21ssh22:20:160
3251997115,43cyclictest0-21swapper/019:50:030
3251997114,42cyclictest0-21swapper/021:44:000
3251997112,45cyclictest0-21swapper/020:17:070
46082700,2sleep10-21swapper/100:13:261
3251997018,38cyclictest26754-21latency_hist20:15:040
3251997014,40cyclictest0-21swapper/022:55:560
3251996915,44cyclictest24502-21/usr/sbin/munin21:20:320
3251996915,44cyclictest24502-21/usr/sbin/munin21:20:320
3251996914,41cyclictest11638-21ssh23:42:260
3251996912,37cyclictest0-21swapper/020:08:490
325299681,6cyclictest29256-21ssh23:25:091
325299681,6cyclictest29256-21ssh23:25:091
325299681,55cyclictest9409-21ntpq19:25:261
3251996818,31cyclictest13489-21ssh21:46:530
3251996811,46cyclictest0-21swapper/021:32:350
325299671,5cyclictest28434-21munin-run20:20:031
3251996717,37cyclictest30772-21ssh22:47:150
3251996715,37cyclictest0-21swapper/019:19:590
325299665,44cyclictest0-21swapper/121:32:571
325299662,44cyclictest332-21ssh22:50:201
3251996619,26cyclictest0-21swapper/000:02:410
3251996619,26cyclictest0-21swapper/000:02:410
3251996616,39cyclictest0-21swapper/019:11:550
3251996615,12cyclictest987-21ssh22:50:290
325299656,36cyclictest0-21swapper/100:34:261
325299655,41cyclictest8050irq/56-dwc_otg_19:20:031
325299651,52cyclictest9035-21sed23:40:031
325299645,42cyclictest7850irq/56-dwc_otg23:21:541
325299645,39cyclictest8050irq/56-dwc_otg_19:55:211
325299643,4cyclictest0-21swapper/121:59:291
325299642,51cyclictest7529-21date19:20:241
3251996414,35cyclictest0-21swapper/020:47:370
3251996414,35cyclictest0-21swapper/020:47:370
3251996413,33cyclictest1837-21/usr/sbin/munin20:35:190
325299635,4cyclictest7850irq/56-dwc_otg21:35:321
325299631,50cyclictest10112-21/usr/sbin/munin23:40:271
325299631,4cyclictest0-21swapper/100:25:551
325299631,46cyclictest8050irq/56-dwc_otg_22:10:191
3251996318,34cyclictest5197-21seq20:44:510
3251996315,33cyclictest10530-21taskset20:58:520
325299628,41cyclictest8050irq/56-dwc_otg_20:14:531
325299624,4cyclictest7850irq/56-dwc_otg23:45:301
325299624,4cyclictest0-21swapper/121:30:031
325299622,43cyclictest31720-21unixbench_singl20:25:321
3251996215,30cyclictest1390-21chrt20:33:440
175752620,2sleep217577-21modprobe19:50:032
325499619,10cyclictest0-21swapper/300:24:113
325299616,5cyclictest15053-21df21:10:211
325299616,42cyclictest8050irq/56-dwc_otg_23:12:461
325299616,33cyclictest0-21swapper/121:19:591
325299616,33cyclictest0-21swapper/121:19:591
3251996119,11cyclictest6905-21dpkg19:20:140
3251996116,29cyclictest0-21swapper/019:36:070
3251996116,29cyclictest0-21swapper/019:36:070
325299603,4cyclictest0-21swapper/123:09:371
325299603,46cyclictest0-21swapper/122:15:341
325299602,5cyclictest10499-21irqstats00:20:231
325299602,5cyclictest0-21swapper/120:50:041
325299602,5cyclictest0-21swapper/120:50:041
325299602,54cyclictest1133-21runrttasks20:00:191
325299601,6cyclictest24261-21sed22:40:041
3251996014,9cyclictest0-21swapper/020:20:130
325299596,41cyclictest8050irq/56-dwc_otg_00:07:401
325299596,39cyclictest7850irq/56-dwc_otg23:30:301
325299592,44cyclictest8050irq/56-dwc_otg_22:48:081
325299591,5cyclictest8717-21df_inode23:00:211
3251995938,11cyclictest18144-21cut19:50:220
325299581,43cyclictest9771-21cat00:20:031
325299574,4cyclictest7950irq/56-dwc_otg_21:41:421
325299572,5cyclictest24906-21ssh22:00:301
325299564,5cyclictest0-21swapper/120:20:221
325299562,6cyclictest31252-21runrttasks22:09:121
325299562,42cyclictest26032-21ssh00:00:191
325299562,42cyclictest26032-21ssh00:00:191
325299561,4cyclictest22955-21runrttasks23:55:311
325299561,4cyclictest14314-21ssh22:27:201
325299561,48cyclictest24483-21ssh23:19:441
325299561,46cyclictest8187-21fschecks_count22:20:201
325299561,46cyclictest8187-21fschecks_count22:20:201
325499553,43cyclictest8050irq/56-dwc_otg_21:55:143
3254995512,4cyclictest0-21swapper/300:30:283
325499551,20cyclictest1241-21ssh22:50:323
325299553,11cyclictest8050irq/56-dwc_otg_20:55:141
325299552,40cyclictest8050irq/56-dwc_otg_19:44:261
325299551,5cyclictest12615-21unixbench-2d21:45:301
325299551,4cyclictest0-21swapper/123:51:301
325299544,42cyclictest8050irq/56-dwc_otg_22:55:231
325299542,39cyclictest0-21swapper/100:37:061
325299541,5cyclictest26275-21ssh22:41:341
3252995413,37cyclictest12931-21latency19:35:241
3252995413,37cyclictest12931-21latency19:35:241
325499534,4cyclictest7850irq/56-dwc_otg21:10:203
3254995311,4cyclictest0-21swapper/300:29:513
325299537,35cyclictest0-21swapper/121:50:501
325299537,35cyclictest0-21swapper/121:50:501
325299534,4cyclictest0-21swapper/121:23:271
325299534,4cyclictest0-21swapper/121:23:271
325299523,42cyclictest8050irq/56-dwc_otg_19:31:111
29112523,42sleep20-21swapper/219:09:092
325499517,3cyclictest0-21swapper/322:48:393
325499511,5cyclictest0-21swapper/323:50:143
325499511,15cyclictest30906-21ls00:05:333
325499502,14cyclictest0-21swapper/321:27:533
325299509,38cyclictest18310-21/usr/sbin/munin19:50:241
325299506,36cyclictest8050irq/56-dwc_otg_20:06:361
325299503,4cyclictest9450irq/48-DMA10:20:151
325299501,5cyclictest8689-21taskset20:53:221
325299501,4cyclictest0-21swapper/119:47:091
325399493,5cyclictest7850irq/56-dwc_otg19:30:162
325299492,5cyclictest0-21swapper/120:35:191
325499489,4cyclictest0-21swapper/323:24:163
325299481,6cyclictest3903-21fschecks_count20:40:201
325299481,6cyclictest32470-21cut20:30:171
325299481,4cyclictest8050irq/56-dwc_otg_19:15:031
29242473,36sleep00-21swapper/019:09:200
325399462,6cyclictest8050irq/56-dwc_otg_20:00:132
325499457,23cyclictest0-21swapper/323:40:173
325499447,22cyclictest0-21swapper/320:10:163
3253994421,6cyclictest17407-21unixbench_singl22:30:322
34942430,3sleep33493-21latency_hist22:15:033
325499437,7cyclictest0-21swapper/322:15:213
240772430,2sleep20-21swapper/223:19:002
325499427,24cyclictest31477-21ssh21:30:163
3254994215,10cyclictest8050irq/56-dwc_otg_23:20:043
325399427,13cyclictest28527-21ssh22:45:162
325399426,13cyclictest7850irq/56-dwc_otg00:05:192
325399424,6cyclictest0-21swapper/219:55:172
325499418,24cyclictest0-21swapper/322:30:173
325399412,12cyclictest0-21swapper/220:15:152
3253994115,5cyclictest14525-21seq23:45:312
325399411,4cyclictest7950irq/56-dwc_otg_23:20:302
3253994110,6cyclictest7850irq/56-dwc_otg22:25:312
243812410,3sleep231-21ksoftirqd/223:58:052
325499408,4cyclictest0-21swapper/323:25:183
325499408,4cyclictest0-21swapper/323:25:183
3254994022,4cyclictest7850irq/56-dwc_otg00:00:223
3254994022,4cyclictest7850irq/56-dwc_otg00:00:223
325399409,4cyclictest0-21swapper/221:47:492
325399407,23cyclictest0-21swapper/223:30:182
325399403,12cyclictest0-21swapper/200:20:142
100752400,18sleep28050irq/56-dwc_otg_23:01:042
325399397,24cyclictest0-21swapper/221:10:092
325399397,23cyclictest0-21swapper/223:25:082
325399397,23cyclictest0-21swapper/223:25:082
325399397,21cyclictest0-21swapper/219:35:172
325399397,21cyclictest0-21swapper/219:35:172
3254993826,7cyclictest381ktimersoftd/322:40:143
325499381,32cyclictest0-21swapper/323:00:163
325399387,22cyclictest0-21swapper/219:50:182
3253993819,5cyclictest16832-21sh21:50:332
3253993819,5cyclictest16832-21sh21:50:332
325399381,3cyclictest7850irq/56-dwc_otg22:50:172
325399381,19cyclictest5549-21ssh23:35:232
325499378,15cyclictest0-21swapper/321:25:043
325499378,15cyclictest0-21swapper/321:25:043
325499378,14cyclictest0-21swapper/321:20:033
325499378,14cyclictest0-21swapper/321:20:033
325499377,22cyclictest0-21swapper/320:40:173
325499371,3cyclictest0-21swapper/323:05:143
325399375,5cyclictest7850irq/56-dwc_otg22:15:172
325399372,3cyclictest7950irq/56-dwc_otg_19:40:142
3253993713,6cyclictest8970-21fschecks_count19:25:202
3254993618,5cyclictest3749-21sh00:11:513
3253993611,15cyclictest28116-21cpu22:05:182
325399358,14cyclictest8050irq/56-dwc_otg_23:40:272
325399352,2cyclictest0-21swapper/220:55:152
325399351,7cyclictest2961-21grep00:10:312
325399351,18cyclictest0-21swapper/200:05:032
325399351,18cyclictest0-21swapper/200:05:032
325499348,6cyclictest0-21swapper/321:35:073
325499348,4cyclictest0-21swapper/323:45:163
325499347,16cyclictest0-21swapper/319:15:043
325499341,4cyclictest0-21swapper/320:05:163
325499341,23cyclictest4919-21ls22:55:223
3254993411,6cyclictest37-21rcuc/319:50:323
325399345,3cyclictest7850irq/56-dwc_otg22:40:302
325399341,5cyclictest7541-21df21:40:172
3253993413,15cyclictest31-21ksoftirqd/200:30:142
273892340,5sleep3325499cyclictest22:04:413
19442340,5sleep2325399cyclictest20:35:192
325499334,23cyclictest8050irq/56-dwc_otg_22:25:143
325499332,21cyclictest0-21swapper/320:26:333
325499332,15cyclictest8050irq/56-dwc_otg_22:20:203
325499332,15cyclictest8050irq/56-dwc_otg_22:20:203
3254993310,11cyclictest8050irq/56-dwc_otg_22:05:183
325399338,5cyclictest7850irq/56-dwc_otg00:15:222
325399338,5cyclictest0-21swapper/221:20:172
325399338,5cyclictest0-21swapper/221:20:172
325399337,5cyclictest13149-21ls23:05:212
325399335,14cyclictest7950irq/56-dwc_otg_22:20:232
325399335,14cyclictest7950irq/56-dwc_otg_22:20:232
325399332,7cyclictest7950irq/56-dwc_otg_22:35:232
325399332,22cyclictest8050irq/56-dwc_otg_21:30:152
3253993313,5cyclictest19721-21aten_rbpower_cu21:55:172
325399331,26cyclictest0-21swapper/221:15:182
325399331,26cyclictest0-21swapper/221:15:182
325399331,23cyclictest22772-21uniq00:35:222
325499329,4cyclictest0-21swapper/300:15:173
325499321,7cyclictest2342-21uptime23:30:333
3254993213,5cyclictest25456-21latency_hist00:00:043
3254993213,10cyclictest7950irq/56-dwc_otg_22:35:233
325399326,13cyclictest8050irq/56-dwc_otg_19:25:042
325399324,5cyclictest17091-21grep23:10:202
325399323,14cyclictest0-21swapper/221:35:222
325399322,4cyclictest7850irq/56-dwc_otg19:10:322
3253993211,4cyclictest0-21swapper/222:55:312
325499317,15cyclictest0-21swapper/319:30:113
325499316,10cyclictest8050irq/56-dwc_otg_21:50:033
325499313,18cyclictest1837-21/usr/sbin/munin20:35:303
325499312,4cyclictest22596-21wget00:35:193
325499311,5cyclictest8050irq/56-dwc_otg_21:05:213
325499311,19cyclictest16632-21timerandwakeup21:50:313
325499311,19cyclictest16632-21timerandwakeup21:50:313
325499311,15cyclictest8050irq/56-dwc_otg_19:25:043
325399317,5cyclictest7950irq/56-dwc_otg_20:25:142
325399317,14cyclictest0-21swapper/221:00:092
325399313,13cyclictest8050irq/56-dwc_otg_22:00:302
325399312,6cyclictest18828-21grep23:50:312
325399311,25cyclictest8946-21ls20:55:032
325399311,22cyclictest6847-21ls19:20:042
28482318,13sleep10-21swapper/119:08:191
325499308,4cyclictest0-21swapper/321:41:343
325499302,4cyclictest7950irq/56-dwc_otg_20:30:253
3254993023,5cyclictest19628-21ssh23:13:253
3254993013,4cyclictest17020-21kworker/u8:419:45:023
325399306,12cyclictest8050irq/56-dwc_otg_20:05:192
325399305,5cyclictest7850irq/56-dwc_otg21:25:302
325399304,4cyclictest13778-21apt-get00:25:092
3253993010,4cyclictest7950irq/56-dwc_otg_20:35:032
325499292,5cyclictest0-21swapper/319:35:133
325499292,5cyclictest0-21swapper/319:35:133
3254992922,5cyclictest30330-21latency_hist20:25:043
325499291,17cyclictest12552-21sh21:05:033
325399299,13cyclictest0-21swapper/222:13:272
325399293,15cyclictest0-21swapper/220:40:082
325399291,8cyclictest5711-21fschecks_count20:45:202
325399291,8cyclictest5711-21fschecks_count20:45:202
325399291,5cyclictest0-21swapper/221:05:262
325399291,15cyclictest26611-21latency_hist20:15:042
325499282,5cyclictest21541-21grep20:00:163
325499282,14cyclictest0-21swapper/320:15:293
325399281,11cyclictest7850irq/56-dwc_otg20:20:132
325499272,5cyclictest8050irq/56-dwc_otg_19:15:173
325499272,12cyclictest8050irq/56-dwc_otg_19:25:183
28452273,16sleep31165-21snmpd19:08:173
3254992620,4cyclictest381ktimersoftd/320:55:183
3254992617,7cyclictest5706-21fschecks_count20:45:203
3254992617,7cyclictest5706-21fschecks_count20:45:203
325499261,5cyclictest5179-21ssh23:35:173
325499252,15cyclictest7941-21ntp_kernel_err20:50:263
325499251,18cyclictest0-21swapper/319:49:103
325499251,12cyclictest8050irq/56-dwc_otg_19:56:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional