You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Mon Jun 30, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87859910431,37cyclictest0-21swapper/008:00:260
87859910431,37cyclictest0-21swapper/008:00:260
323742970,5sleep1878699cyclictest08:15:231
115752860,2sleep30-21swapper/311:57:183
878599857,25cyclictest0-21swapper/009:45:270
320752800,5sleep2878799cyclictest12:22:092
878599792,38cyclictest0-21swapper/011:35:260
315572790,2sleep337-21rcuc/310:12:593
8786997744,12cyclictest23-21ksoftirqd/110:45:251
878599749,53cyclictest975-21missed_timers10:15:370
878599749,53cyclictest975-21missed_timers10:15:370
878599738,48cyclictest13691-21unixbench_multi08:50:430
131082730,1sleep00-21swapper/012:00:090
878599697,48cyclictest11303-21ntpq07:15:340
878599697,46cyclictest16308-21unixbench-2d09:50:440
878599695,50cyclictest10995-21taskset11:15:320
8785996928,3cyclictest0-21swapper/008:25:240
878599691,54cyclictest22652-21grep10:45:430
878799672,51cyclictest0-21swapper/210:40:322
8787996723,26cyclictest30138-21apt-get09:25:162
878599676,58cyclictest0-21swapper/010:55:160
878599675,50cyclictest6342-21timerandwakeup11:50:420
197332670,3sleep10-21swapper/109:55:381
878599669,33cyclictest0-21swapper/008:35:250
878599661,52cyclictest29434-21cpu10:10:290
8785996614,23cyclictest22384-21sh10:00:260
878599658,12cyclictest0-21swapper/007:30:270
195532650,2sleep00-21swapper/011:27:200
8785996420,26cyclictest21084-21munin-plugin-st10:45:100
878599641,48cyclictest24053-21/usr/sbin/munin07:50:390
293752640,2sleep00-21swapper/012:19:450
878799636,45cyclictest7950irq/56-dwc_otg_07:10:272
878599635,33cyclictest0-21swapper/009:15:270
878599631,32cyclictest19131-21grep09:05:430
878899624,15cyclictest7950irq/56-dwc_otg_12:20:283
878799629,44cyclictest8050irq/56-dwc_otg_11:15:372
878799621,6cyclictest19016-21timerwakeupswit09:05:422
878599622,29cyclictest2881-21rm12:25:400
878799615,4cyclictest8050irq/56-dwc_otg_10:15:442
878799615,4cyclictest8050irq/56-dwc_otg_10:15:442
878599611,34cyclictest0-21swapper/011:20:250
878799601,4cyclictest7850irq/56-dwc_otg10:10:162
878799598,40cyclictest0-21swapper/212:16:022
878799597,4cyclictest0-21swapper/211:57:462
8787995910,34cyclictest0-21swapper/209:45:082
878599599,35cyclictest18832-21switchtime07:35:360
8785995938,8cyclictest0-21swapper/009:25:210
878599592,45cyclictest0-21swapper/007:43:400
878599590,48cyclictest32423-21runrttasks08:15:260
878799581,3cyclictest7950irq/56-dwc_otg_11:10:342
878599586,39cyclictest0-21swapper/008:30:260
878599584,9cyclictest0-21swapper/009:35:280
878599584,9cyclictest0-21swapper/009:35:280
8785995832,5cyclictest7850irq/56-dwc_otg10:05:400
8785995831,9cyclictest101ktimersoftd/011:10:280
878599582,47cyclictest0-21swapper/007:55:310
878599581,54cyclictest17757-21idleruntime-cro09:05:090
8787995712,42cyclictest0-21swapper/207:40:252
8785995742,8cyclictest0-21swapper/012:20:430
83982573,47sleep10-21swapper/107:08:371
83982573,47sleep10-21swapper/107:08:371
878799563,10cyclictest7850irq/56-dwc_otg11:04:202
878799563,10cyclictest7850irq/56-dwc_otg11:04:202
878599568,38cyclictest0-21swapper/012:00:240
8785995633,15cyclictest6308-21cut12:30:330
8785995625,6cyclictest0-21swapper/010:25:280
878599561,29cyclictest0-21swapper/009:20:390
878599561,29cyclictest0-21swapper/009:20:390
878799554,4cyclictest7850irq/56-dwc_otg08:00:272
878799554,4cyclictest7850irq/56-dwc_otg08:00:272
878799554,11cyclictest7850irq/56-dwc_otg11:29:072
878799551,4cyclictest0-21swapper/209:58:212
878799551,44cyclictest0-21swapper/212:39:592
878799551,11cyclictest7850irq/56-dwc_otg11:37:372
878599558,35cyclictest14197-21latency_hist10:35:100
878599558,23cyclictest0-21swapper/007:10:200
878599555,46cyclictest101ktimersoftd/010:51:100
878599555,20cyclictest24975-21dpkg11:35:100
8785995550,3cyclictest0-21swapper/010:22:520
8785995534,10cyclictest16576-21runrttasks10:37:430
878599552,38cyclictest20302-21ssh09:56:220
878799544,4cyclictest13863-21/usr/sbin/munin12:00:292
878799542,4cyclictest0-21swapper/210:02:382
878799542,3cyclictest7950irq/56-dwc_otg_09:45:272
8785995435,8cyclictest101ktimersoftd/011:45:350
162272540,2sleep30-21swapper/311:23:123
878799539,4cyclictest9471-21kernelversion08:40:342
878799539,4cyclictest0-21swapper/210:23:332
878799536,7cyclictest8050irq/56-dwc_otg_11:48:082
878799536,3cyclictest7850irq/56-dwc_otg11:09:562
878799532,10cyclictest6492-21/usr/sbin/munin08:30:402
8787995314,4cyclictest13056-21kernelversion08:50:342
878799531,3cyclictest0-21swapper/211:42:552
878799531,3cyclictest0-21swapper/207:35:212
878599533,29cyclictest134-21systemd-journal11:05:100
878599533,29cyclictest134-21systemd-journal11:05:100
8785995332,10cyclictest0-21swapper/011:42:350
878599532,39cyclictest11829-21ssh12:36:500
8788995244,5cyclictest22378-21ssh10:00:263
878799527,3cyclictest8050irq/56-dwc_otg_10:29:302
878799524,3cyclictest8050irq/56-dwc_otg_10:45:262
878799523,4cyclictest8050irq/56-dwc_otg_12:34:122
878799522,5cyclictest25951-21timerwakeupswit10:50:422
878799521,8cyclictest8050irq/56-dwc_otg_09:19:442
878799521,12cyclictest0-21swapper/211:32:122
8787995210,37cyclictest6500-21unixbench_multi11:50:432
878599528,40cyclictest101ktimersoftd/009:11:240
8785995232,12cyclictest0-21swapper/009:34:160
8785995231,4cyclictest101ktimersoftd/011:10:070
878799518,4cyclictest0-21swapper/207:30:312
878799515,4cyclictest8050irq/56-dwc_otg_10:05:352
878799515,3cyclictest7950irq/56-dwc_otg_11:21:212
878799514,3cyclictest8050irq/56-dwc_otg_12:10:292
878799514,3cyclictest8050irq/56-dwc_otg_12:10:292
878799511,6cyclictest25457-21cut07:55:322
878799511,5cyclictest331-21/usr/sbin/munin08:15:312
878799511,5cyclictest22621-21unixbench-2d07:45:402
878799511,4cyclictest28495-21apt-get10:55:232
878799511,4cyclictest19823-21ssh12:07:212
8787995114,35cyclictest11336-21sh10:30:342
878799511,3cyclictest0-21swapper/209:51:392
878799511,2cyclictest0-21swapper/212:28:242
878799511,10cyclictest0-21swapper/209:21:162
878799511,10cyclictest0-21swapper/209:21:162
878699518,27cyclictest0-21swapper/108:55:291
878699518,27cyclictest0-21swapper/108:55:291
878699517,27cyclictest0-21swapper/109:25:161
878699514,4cyclictest7950irq/56-dwc_otg_12:25:241
8785995135,8cyclictest0-21swapper/012:15:020
8785995135,8cyclictest0-21swapper/012:15:020
8785995127,11cyclictest0-21swapper/007:46:340
8785995122,8cyclictest31737-21unixbench_singl08:10:380
878899509,11cyclictest1054-21ssh11:45:193
878799506,5cyclictest15025-21ls10:35:352
878799502,8cyclictest24486-21unixbench_multi07:50:452
878799502,7cyclictest0-21swapper/208:45:202
878599506,24cyclictest8784-21cyclictest09:40:400
8785995030,4cyclictest0-21swapper/008:40:420
8785995023,15cyclictest0-21swapper/008:20:250
878599501,26cyclictest18776-21grep12:05:450
878899498,26cyclictest0-21swapper/308:25:263
878899497,27cyclictest0-21swapper/309:25:263
878799499,4cyclictest8460-21taskset08:36:562
878799498,6cyclictest3664-21basename08:25:272
878799496,5cyclictest8050irq/56-dwc_otg_09:11:082
878799492,8cyclictest32123-21modprobe08:15:092
878799491,8cyclictest11678-21grep07:15:402
878799491,3cyclictest7850irq/56-dwc_otg09:30:372
878799491,3cyclictest0-21swapper/209:03:252
878699496,11cyclictest7950irq/56-dwc_otg_07:20:211
8785994923,5cyclictest12533-21/usr/sbin/munin07:20:310
878899481,8cyclictest8050irq/56-dwc_otg_09:30:303
878799484,6cyclictest0-21swapper/207:25:102
878799481,8cyclictest5963-21grep09:35:462
878799481,8cyclictest5963-21grep09:35:462
878799481,8cyclictest15056-21ntp_kernel_err08:55:392
878799481,8cyclictest15056-21ntp_kernel_err08:55:392
8785994817,20cyclictest0-21swapper/008:06:300
878599481,33cyclictest0-21swapper/008:58:320
878599481,33cyclictest0-21swapper/008:58:320
82782483,37sleep00-21swapper/007:07:020
82782483,37sleep00-21swapper/007:07:020
878899479,19cyclictest6529-21unixbench_singl11:50:433
878899477,25cyclictest0-21swapper/307:35:233
878799478,4cyclictest30122-21sshd08:07:402
878799472,4cyclictest0-21swapper/207:29:402
878699477,27cyclictest0-21swapper/107:10:231
878699473,8cyclictest0-21swapper/111:20:261
878899468,25cyclictest0-21swapper/308:10:233
878899467,24cyclictest0-21swapper/307:40:253
8788994618,6cyclictest21999-21latency_hist10:00:103
878599461,25cyclictest14833-21latency07:25:300
878899457,24cyclictest0-21swapper/311:35:153
878899447,23cyclictest0-21swapper/311:25:273
878899447,23cyclictest0-21swapper/311:00:263
878899447,23cyclictest0-21swapper/311:00:263
878899445,22cyclictest8050irq/56-dwc_otg_07:15:393
878899445,19cyclictest25189-21/usr/sbin/munin07:55:283
878899442,12cyclictest26460-21ssh09:19:383
8788994412,16cyclictest8836-21/usr/sbin/munin07:10:233
878699445,28cyclictest7850irq/56-dwc_otg09:45:271
8785994415,18cyclictest0-21swapper/008:47:070
878899437,8cyclictest32382-21apt-get08:15:243
878899437,26cyclictest0-21swapper/310:30:273
878899437,22cyclictest0-21swapper/307:20:153
878899437,14cyclictest21053-21ssh12:09:383
878899434,11cyclictest1121-21ssh10:15:393
878899434,11cyclictest1121-21ssh10:15:393
8788994312,12cyclictest18736-21ssh10:40:423
8788994311,11cyclictest26388-21ssh10:51:153
878699436,9cyclictest0-21swapper/110:20:271
878899428,19cyclictest7950irq/56-dwc_otg_11:31:463
8788994211,17cyclictest13454-21cat08:50:403
8788994211,13cyclictest12437-21ssh09:45:403
8788994210,12cyclictest28861-21munin-plugin-st10:10:093
175242420,3sleep30-21swapper/310:40:043
878899419,12cyclictest9012-21ssh11:12:493
878899413,29cyclictest8050irq/56-dwc_otg_09:40:293
8788994122,8cyclictest8050irq/56-dwc_otg_12:37:123
8788994111,22cyclictest12857-21ssh11:18:133
878699412,8cyclictest7950irq/56-dwc_otg_07:30:251
8786994114,10cyclictest7850irq/56-dwc_otg10:35:111
229952410,4sleep122993-21grep10:00:371
878899407,17cyclictest31572-21rm11:42:123
8788994024,10cyclictest7950irq/56-dwc_otg_09:40:023
8788994024,10cyclictest7950irq/56-dwc_otg_09:40:023
8788994010,12cyclictest10261-21ssh10:29:223
878699405,5cyclictest0-21swapper/111:50:261
8786994014,6cyclictest4168-21sh09:34:401
217312400,2sleep30-21swapper/310:45:323
878899395,12cyclictest5359-21ssh10:21:083
878899395,11cyclictest8935-21ssh12:34:053
8788993913,8cyclictest6057-21ntp_states08:30:343
8788993910,17cyclictest28256-21ssh12:17:433
878699396,25cyclictest0-21swapper/109:10:271
878699395,24cyclictest955-21runrttasks08:25:241
878699394,5cyclictest7950irq/56-dwc_otg_10:15:431
878699394,5cyclictest7950irq/56-dwc_otg_10:15:431
878699393,3cyclictest0-21swapper/107:40:251
8786993918,5cyclictest10761-21latency_hist10:30:101
878899384,19cyclictest4554-21ssh12:28:373
878899383,4cyclictest0-21swapper/308:05:233
878899383,13cyclictest21364-21ssh09:11:043
878699388,26cyclictest0-21swapper/111:35:161
878699387,21cyclictest0-21swapper/107:15:251
878699385,20cyclictest10395-21ssh11:15:241
8786993818,5cyclictest1047-21sh12:24:451
878699381,5cyclictest29603-21munin-run12:20:091
8786993815,4cyclictest29401-21aten_rbpower_vo10:10:291
8786993815,12cyclictest11195-21iwlist08:45:301
8786993810,6cyclictest7850irq/56-dwc_otg10:35:441
878899378,17cyclictest0-21swapper/309:05:273
878899378,16cyclictest25394-21ssh12:15:023
878899378,16cyclictest25394-21ssh12:15:023
878899375,21cyclictest0-21swapper/309:55:043
878899372,4cyclictest8050irq/56-dwc_otg_10:55:253
878899372,21cyclictest8050irq/56-dwc_otg_12:00:263
878799371,7cyclictest3009-21runrttasks08:21:232
878699377,5cyclictest7850irq/56-dwc_otg11:08:501
878699377,26cyclictest0-21swapper/109:35:301
878699377,26cyclictest0-21swapper/109:35:301
878699377,26cyclictest0-21swapper/109:00:271
878699377,22cyclictest0-21swapper/111:55:261
878699377,22cyclictest0-21swapper/108:40:271
878699376,6cyclictest7850irq/56-dwc_otg11:40:401
878699373,20cyclictest19583-21ssh09:10:101
878699373,20cyclictest0-21swapper/110:55:161
878699371,26cyclictest10310-21ls12:35:321
878699371,20cyclictest31516-21timerandwakeup08:10:371
878699371,19cyclictest0-21swapper/112:07:261
878699365,4cyclictest25973-21/usr/sbin/munin10:05:261
878699363,26cyclictest7850irq/56-dwc_otg12:10:141
878699363,26cyclictest7850irq/56-dwc_otg12:10:141
878699362,23cyclictest0-21swapper/110:40:281
878899358,24cyclictest0-21swapper/308:40:273
878899357,14cyclictest8050irq/56-dwc_otg_07:30:423
878899351,21cyclictest23654-21grep07:50:333
878699359,7cyclictest14781-21ssh12:00:401
878699355,6cyclictest7850irq/56-dwc_otg11:25:281
878699354,21cyclictest0-21swapper/111:45:251
878699354,21cyclictest0-21swapper/107:50:331
8786993516,9cyclictest8737-21latency_hist08:40:101
8786993512,5cyclictest7850irq/56-dwc_otg09:25:101
8786993512,5cyclictest7850irq/56-dwc_otg09:25:101
878699351,24cyclictest29294-21latency08:05:321
8786993511,4cyclictest0-21swapper/109:20:111
8786993510,12cyclictest0-21swapper/112:30:241
878899349,11cyclictest0-21swapper/308:21:413
878899346,15cyclictest0-21swapper/307:25:243
8788993417,8cyclictest7850irq/56-dwc_otg08:03:133
8788993417,8cyclictest7850irq/56-dwc_otg08:03:133
8788993415,7cyclictest8050irq/56-dwc_otg_11:10:073
878899341,18cyclictest29103-21rm09:22:293
878899341,18cyclictest29103-21rm09:22:293
878699348,5cyclictest0-21swapper/111:14:331
878699345,7cyclictest7950irq/56-dwc_otg_08:25:091
878699345,5cyclictest7850irq/56-dwc_otg08:30:281
878699341,24cyclictest27547-21ls08:00:351
878699341,24cyclictest27547-21ls08:00:351
878699341,23cyclictest21826-21cut07:45:301
8786993410,4cyclictest7850irq/56-dwc_otg11:00:381
8786993410,4cyclictest7850irq/56-dwc_otg11:00:381
878899337,11cyclictest8050irq/56-dwc_otg_07:45:273
878899332,17cyclictest0-21swapper/308:36:073
8788993312,7cyclictest16408-21df_inode09:00:293
878699334,21cyclictest7850irq/56-dwc_otg09:50:311
878899323,16cyclictest8050irq/56-dwc_otg_08:45:273
878699321,17cyclictest7850irq/56-dwc_otg07:59:161
8786993211,5cyclictest7850irq/56-dwc_otg09:40:261
8786993210,5cyclictest7950irq/56-dwc_otg_11:30:231
878699316,15cyclictest0-21swapper/108:50:271
8786993116,4cyclictest24831-21/usr/sbin/munin10:50:411
76802305,10sleep30-21swapper/307:05:363
76802305,10sleep30-21swapper/307:05:363
8788992920,6cyclictest14397-21apt-get08:55:273
8788992920,6cyclictest14397-21apt-get08:55:273
878699293,4cyclictest7950irq/56-dwc_otg_07:35:301
878699291,15cyclictest0-21swapper/107:25:391
79212260,2sleep20-21swapper/207:05:382
79212260,2sleep20-21swapper/207:05:382
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional