You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Jul 04, 2025 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1313122590,2sleep30-21swapper/319:05:323
231762980,2sleep30-21swapper/322:31:333
1451399943,6cyclictest14890-21fschecks_time23:45:290
1451399902,3cyclictest0-21swapper/019:30:400
1451399891,9cyclictest0-21swapper/019:50:130
1451399891,9cyclictest0-21swapper/019:50:130
1451399891,6cyclictest3840-21unixbench_singl22:05:410
1451399891,5cyclictest2587-21munin-run23:30:110
1451399881,73cyclictest0-21swapper/019:55:250
14513998636,43cyclictest7950irq/56-dwc_otg_22:00:190
14513998611,62cyclictest9250-21sh22:14:020
1451399842,7cyclictest0-21swapper/022:58:590
1451399841,74cyclictest134-21systemd-journal00:28:080
1451399841,67cyclictest1823-21unixbench_singl22:45:420
1451399841,67cyclictest11918-21grep23:40:420
1451399831,69cyclictest30667-21unixbench_multi21:15:420
1451399831,69cyclictest30667-21unixbench_multi21:15:420
14513998129,40cyclictest8050irq/56-dwc_otg_23:35:280
14513998129,40cyclictest8050irq/56-dwc_otg_23:35:280
1451399812,7cyclictest0-21swapper/023:51:090
66252800,5sleep11451699cyclictest00:17:271
1451399801,5cyclictest4029-21grep20:10:320
14513997912,54cyclictest30989-21cut22:00:120
1451799783,21cyclictest25662-21ssh21:10:262
14513997830,38cyclictest19733-21latency_hist19:25:110
14513997829,36cyclictest3644-21grep23:30:340
14513997814,52cyclictest19028-21sh23:09:230
1451399776,56cyclictest0-21swapper/021:50:260
1451399768,55cyclictest23222-21ssh23:14:590
1451399762,8cyclictest0-21swapper/000:02:020
1451399762,6cyclictest12754-21cut21:35:350
14513997510,55cyclictest0-21swapper/021:30:240
1451399741,60cyclictest0-21swapper/021:26:280
14513997412,43cyclictest18282-21latency_hist20:50:120
14513997411,48cyclictest22873-21ssh21:49:180
14513997411,48cyclictest22873-21ssh21:49:180
14516997214,42cyclictest28146-21sh00:02:071
1451399728,47cyclictest3782-21ssh21:24:080
1451399722,6cyclictest955-21runrttasks21:40:530
1451399722,6cyclictest11107-21wget20:30:270
14513997221,39cyclictest30317-21unixbench-2d22:40:450
1451399721,6cyclictest8076-21/usr/sbin/munin00:20:360
1451399721,60cyclictest0-21swapper/022:17:240
1451399721,5cyclictest0-21swapper/020:30:040
1451699711,13cyclictest7950irq/56-dwc_otg_23:30:361
1451399711,54cyclictest21112-21ssh00:37:590
1451399711,54cyclictest21112-21ssh00:37:590
1451399702,7cyclictest0-21swapper/000:09:380
14513997027,31cyclictest8050irq/56-dwc_otg_21:10:250
14513997020,34cyclictest7850irq/56-dwc_otg22:30:380
1451399701,55cyclictest30180-21vmstat19:50:410
1451399701,54cyclictest21133-21ssh22:30:100
14513997011,48cyclictest20193-21/usr/sbin/munin19:25:310
1451699697,25cyclictest0-21swapper/122:30:271
1451399699,50cyclictest0-21swapper/023:56:320
1451399697,48cyclictest27281-21ssh23:20:100
1451399692,6cyclictest3547-21latency_hist20:10:110
1451399691,5cyclictest0-21swapper/023:24:110
1451399691,51cyclictest5853-21taskset22:51:020
179162680,2sleep30-21swapper/321:41:563
1451399682,47cyclictest24063-21fschecks_time21:05:320
14513996810,46cyclictest17235-21runrttasks22:24:520
14513996810,46cyclictest15147-21ssh23:04:210
14513996810,46cyclictest15147-21ssh23:04:210
143052680,2sleep30-21swapper/323:02:373
143052680,2sleep30-21swapper/323:02:373
1451699675,46cyclictest7950irq/56-dwc_otg_20:30:131
1451399671,55cyclictest969-21snmpd22:37:450
1451399662,5cyclictest0-21swapper/020:50:300
14513996619,38cyclictest8050irq/56-dwc_otg_20:41:360
1451399661,52cyclictest8867-21seq20:24:240
1451699654,45cyclictest8050irq/56-dwc_otg_22:00:421
1451399652,7cyclictest0-21swapper/000:15:170
1451399652,7cyclictest0-21swapper/000:12:180
1451399652,5cyclictest0-21swapper/020:55:460
14516996418,43cyclictest0-21swapper/121:30:241
14513996412,38cyclictest0-21swapper/000:31:280
1451399636,4cyclictest508-21in:imuxsock19:17:110
1451399632,4cyclictest0-21swapper/021:04:370
14518996220,25cyclictest29124-21ssh00:04:203
14518996219,28cyclictest0-21swapper/323:40:423
14518996219,21cyclictest12871-21ls00:25:413
1451699623,9cyclictest8050irq/56-dwc_otg_22:40:271
1451399628,39cyclictest0-21swapper/020:35:540
14513996213,34cyclictest16036-21sshd19:14:300
14513996212,31cyclictest8050irq/56-dwc_otg_19:41:560
14518996129,9cyclictest3768-21apt-get20:10:283
1451699613,4cyclictest7950irq/56-dwc_otg_20:15:451
1451399617,41cyclictest0-21swapper/020:01:430
4742600,2sleep30-21swapper/323:26:003
14516996010,31cyclictest0-21swapper/122:39:281
1451699597,35cyclictest0-21swapper/121:50:261
1451699595,4cyclictest7950irq/56-dwc_otg_21:21:471
1451699591,10cyclictest8050irq/56-dwc_otg_20:30:341
1451399591,7cyclictest0-21swapper/019:39:030
14518995832,9cyclictest8090-21apt-get00:20:243
1451699582,4cyclictest0-21swapper/122:50:121
1451699581,4cyclictest7950irq/56-dwc_otg_23:22:411
1451899573,11cyclictest8050irq/56-dwc_otg_19:10:243
14518995728,8cyclictest6798-21apt-get23:35:233
14518995728,8cyclictest6798-21apt-get23:35:233
14518995721,17cyclictest22786-21timerwakeupswit19:30:413
1451699575,2cyclictest0-21swapper/100:20:241
1451699573,7cyclictest0-21swapper/121:46:301
1451699573,7cyclictest0-21swapper/121:46:301
1451699572,4cyclictest0-21swapper/120:40:241
1451699571,11cyclictest27326-21modprobe23:20:111
14513995723,23cyclictest8050irq/56-dwc_otg_20:15:550
14518995623,14cyclictest18091-21munin-run20:50:113
14518995620,14cyclictest3145-21df_inode23:30:283
1451699569,44cyclictest0-21swapper/123:25:221
1451699569,35cyclictest0-21swapper/122:25:511
1451699562,9cyclictest0-21swapper/122:19:121
1451699561,4cyclictest7950irq/56-dwc_otg_20:10:381
14516995614,37cyclictest12481-21fschecks_count21:35:321
14516995614,30cyclictest25298-21diskmemload23:48:571
14518995522,11cyclictest16142-21df23:05:283
14518995521,21cyclictest9132-21rm22:13:473
14518995519,20cyclictest30836-21apt-get22:00:173
14518995519,18cyclictest9622-21ssh22:56:003
1451699551,9cyclictest8050irq/56-dwc_otg_22:08:531
1451699551,9cyclictest15966-21chrt22:22:131
1451699551,6cyclictest18086-21latency_hist20:50:111
1451699551,6cyclictest0-21swapper/123:40:251
1451699551,5cyclictest8050irq/56-dwc_otg_22:11:331
1451699551,44cyclictest0-21swapper/123:08:481
14518995418,24cyclictest20561-21rm22:28:593
14518995418,20cyclictest14138-21sh22:20:283
14518995416,24cyclictest29591-21ssh21:57:333
14518995416,22cyclictest27102-21ssh23:19:483
1451699542,6cyclictest0-21swapper/123:35:231
1451699542,6cyclictest0-21swapper/123:35:231
1451699541,9cyclictest5879-21taskset22:51:051
1451699541,9cyclictest134-21systemd-journal00:38:101
1451699541,9cyclictest134-21systemd-journal00:38:101
14518995321,12cyclictest29927-21ntp_states22:40:403
14518995319,11cyclictest22989-21ls23:55:343
14518995315,23cyclictest14684-21ssh21:38:073
14518995314,13cyclictest1833-21users22:45:423
1451699539,39cyclictest0-21swapper/121:55:241
1451699537,3cyclictest0-21swapper/100:07:471
1451699533,8cyclictest7950irq/56-dwc_otg_23:01:481
1451699533,8cyclictest7950irq/56-dwc_otg_23:01:481
1451699533,8cyclictest22881-21unixbench_multi19:30:421
1451699533,47cyclictest15152-21ssh00:30:121
1451699532,5cyclictest19873-21ssh23:10:241
14516995311,38cyclictest16078-21cut21:40:281
229902520,2sleep222991-21sh21:49:322
229902520,2sleep222991-21sh21:49:322
14518995218,21cyclictest26962-21ssh21:54:503
14518995214,24cyclictest7950irq/56-dwc_otg_22:51:033
1451699521,9cyclictest0-21swapper/123:00:101
1451699521,7cyclictest20138-21apt-get20:55:241
1451699521,4cyclictest0-21swapper/121:15:421
1451699521,4cyclictest0-21swapper/121:15:421
1451699521,3cyclictest0-21swapper/121:26:161
1451699521,10cyclictest18755-21taskset20:50:351
1451699521,10cyclictest0-21swapper/119:25:331
14518995122,12cyclictest20886-21ls23:10:403
14518995121,8cyclictest3530-21munin-run20:10:113
14518995118,18cyclictest29922-21ssh23:22:423
14518995114,25cyclictest20183-21sh19:25:303
1451799511,2cyclictest0-21swapper/200:10:252
1451699517,7cyclictest28284-21grep19:45:421
1451699517,7cyclictest28284-21grep19:45:421
1451699515,4cyclictest7850irq/56-dwc_otg00:33:541
1451699511,8cyclictest0-21swapper/121:14:321
1451699511,5cyclictest20801-21ssh23:52:461
1451699511,4cyclictest8050irq/56-dwc_otg_19:45:121
14516995111,4cyclictest24263-21cat21:05:341
1451699511,10cyclictest3626-21cat20:10:121
259902500,2sleep30-21swapper/322:35:353
1451899509,28cyclictest0-21swapper/321:15:413
1451899509,28cyclictest0-21swapper/321:15:413
14518995021,11cyclictest20196-21ssh23:51:323
14518995019,16cyclictest25298-21diskmemload00:12:573
14518995018,12cyclictest27958-21rm21:12:463
14518995016,22cyclictest4436-21ssh00:15:183
14518995013,16cyclictest3767-21ssh21:24:073
1451699503,4cyclictest0-21swapper/120:40:121
1451699502,7cyclictest8241-21switchtime20:20:391
1451699501,5cyclictest0-21swapper/119:39:061
1451699501,3cyclictest0-21swapper/100:14:361
1451699492,8cyclictest0-21swapper/119:50:401
1451699491,8cyclictest32722-21aten_rbpower_vo20:00:301
1451699491,3cyclictest8050irq/56-dwc_otg_23:56:361
1451699491,10cyclictest0-21swapper/119:25:121
1451899489,23cyclictest0-21swapper/319:47:303
1451899489,23cyclictest0-21swapper/319:47:303
1451899487,31cyclictest0-21swapper/300:33:543
14518994826,14cyclictest8438-21sh21:30:243
1451899479,20cyclictest0-21swapper/322:08:313
1451899477,26cyclictest0-21swapper/320:40:183
14518994716,18cyclictest0-21swapper/322:19:303
1451699472,7cyclictest8050irq/56-dwc_otg_21:01:381
1451699471,5cyclictest32017-21chrt19:56:041
1451699471,4cyclictest8050irq/56-dwc_otg_19:19:441
139042473,12sleep00-21swapper/019:05:460
1451899469,22cyclictest14235-21runrttasks20:38:423
14518994621,9cyclictest18336-21apt-get20:50:293
14518994614,10cyclictest0-21swapper/323:45:583
1451799467,25cyclictest0-21swapper/219:20:282
1451799467,24cyclictest0-21swapper/223:00:192
1451799467,24cyclictest0-21swapper/223:00:192
1451699462,8cyclictest0-21swapper/119:10:381
14518994518,12cyclictest30133-21unixbench_singl19:50:413
14518994517,13cyclictest23023-21ssh21:49:363
14518994517,13cyclictest23023-21ssh21:49:363
1451799457,23cyclictest0-21swapper/220:20:262
14517994526,14cyclictest9550irq/80-mmc020:40:242
1451799442,7cyclictest8050irq/56-dwc_otg_22:05:232
1451799442,7cyclictest0-21swapper/223:25:212
1451799441,13cyclictest29253-21cut19:50:302
141412440,2sleep20-21swapper/222:20:282
14518994318,13cyclictest6465-21ssh21:26:503
14518994316,14cyclictest10753-21ls20:30:123
14518994315,18cyclictest19914-21ls19:25:123
14518994314,21cyclictest32260-21seq19:59:133
14518994313,13cyclictest31845-21ssh00:07:043
14518994311,12cyclictest20558-21ssh00:36:273
14518994311,12cyclictest20558-21ssh00:36:273
1451799437,23cyclictest0-21swapper/221:00:302
261452420,3sleep213916-1kworker/u9:022:35:382
14518994219,12cyclictest11199-21df20:30:283
1451799427,25cyclictest0-21swapper/223:10:172
1451799426,24cyclictest0-21swapper/219:10:262
1451899417,25cyclictest0-21swapper/320:00:293
14518994112,12cyclictest0-21swapper/320:22:013
14517994115,6cyclictest7850irq/56-dwc_otg00:40:132
14517994115,6cyclictest7850irq/56-dwc_otg00:40:132
1451899406,11cyclictest955-21runrttasks20:18:443
14518994014,14cyclictest21595-21chrt20:59:053
1451799407,20cyclictest0-21swapper/220:15:282
14517994022,5cyclictest24011-21iwlist19:35:312
219372390,2sleep221940-21cpuspeed19:30:292
14518993912,14cyclictest0-21swapper/319:39:303
14518993912,12cyclictest8050irq/56-dwc_otg_21:10:043
1451799393,28cyclictest8050irq/56-dwc_otg_20:50:292
1451799391,9cyclictest3582-21ls23:30:332
1451799391,29cyclictest10937-21ls20:30:132
14517993912,19cyclictest25001-21latency_hist22:35:122
14518993823,10cyclictest26200-21/usr/sbin/munin19:40:393
1451799383,18cyclictest31-21ksoftirqd/220:05:252
14517993824,11cyclictest0-21swapper/221:40:242
1451799381,9cyclictest31-21ksoftirqd/221:15:312
1451799381,9cyclictest31-21ksoftirqd/221:15:312
14517993813,5cyclictest11074-21cut20:30:272
1451799381,25cyclictest27449-21cut19:45:322
1451799381,25cyclictest27449-21cut19:45:322
14518993714,9cyclictest0-21swapper/319:17:123
1451799374,4cyclictest24076-21sshd23:15:292
1451799374,18cyclictest0-21swapper/221:05:422
1451799373,5cyclictest17178-21grep23:05:412
14517993714,10cyclictest0-21swapper/220:35:392
1451799371,25cyclictest12432-21ls21:35:302
14518993610,14cyclictest0-21swapper/321:01:563
1451799369,7cyclictest9260-21grep00:20:412
1451799369,5cyclictest7850irq/56-dwc_otg21:30:372
1451799362,6cyclictest27344-21runrttasks21:55:122
1451799362,5cyclictest5165-21ntp_states22:50:382
14517993621,4cyclictest2752550wget21:55:272
14517993617,4cyclictest25298-21diskmemload21:20:312
1451799357,4cyclictest0-21swapper/222:50:122
1451799357,25cyclictest0-21swapper/220:00:292
1451799355,8cyclictest7850irq/56-dwc_otg22:40:382
1451799354,5cyclictest22294-21ssh23:55:232
1451799354,4cyclictest29-21rcuc/220:55:412
14517993514,10cyclictest31-21ksoftirqd/200:25:312
1451799351,18cyclictest0-21swapper/200:15:402
1451799351,15cyclictest0-21swapper/219:25:272
107842350,2sleep20-21swapper/222:58:292
102782350,2sleep20-21swapper/223:40:042
102782350,2sleep20-21swapper/223:40:042
1451799342,17cyclictest26700-21latency00:00:342
1451799342,15cyclictest0-21swapper/223:50:242
14517993416,5cyclictest10603-21if_err_wlan022:15:322
1451799341,23cyclictest17292-21timerandwakeup19:15:392
1451799335,17cyclictest7950irq/56-dwc_otg_00:30:252
1451799331,4cyclictest5290-21ssh21:25:332
1451799331,23cyclictest16819-21uniq20:45:362
1451799326,7cyclictest7950irq/56-dwc_otg_23:40:312
1451799324,7cyclictest3204550ssh22:00:372
1451799323,15cyclictest7950irq/56-dwc_otg_22:25:242
1451799322,19cyclictest7850irq/56-dwc_otg23:20:282
1451799321,21cyclictest8308-21rm22:12:062
1451799316,6cyclictest7950irq/56-dwc_otg_19:40:352
1451799314,8cyclictest31-21ksoftirqd/223:48:242
1451799311,6cyclictest29863-21apt-get00:05:182
1451799308,11cyclictest32259-21taskset19:59:122
1451799292,18cyclictest0-21swapper/220:10:362
127442296,10sleep20-21swapper/219:05:282
126852297,14sleep10-21swapper/119:05:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional