You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-17 - 09:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Oct 17, 2025 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
489521180,8sleep12297799cyclictest00:10:171
22976999825,55cyclictest3710-21latency_hist19:50:170
2297699968,16cyclictest0-21swapper/019:10:200
2297699968,16cyclictest0-21swapper/019:10:200
224352880,2sleep10-21swapper/122:25:331
244972860,2sleep20-21swapper/223:10:312
2297699853,62cyclictest20207-21cron23:05:170
2297699842,71cyclictest0-21swapper/023:10:180
2297699841,65cyclictest26189-21ps19:20:380
2297699837,26cyclictest0-21swapper/019:40:230
125412820,6sleep1221ktimersoftd/100:20:321
2297699811,28cyclictest4536-21aten_rbpower_po21:15:360
2297699811,15cyclictest0-21swapper/000:20:170
153992800,2sleep00-21swapper/000:24:530
2297699791,21cyclictest5726-21fschecks_time22:45:370
259322760,2sleep20-21swapper/221:47:172
2297699766,41cyclictest0-21swapper/020:35:360
22976997519,5cyclictest0-21swapper/020:25:390
2297699751,26cyclictest3192-21aten_rbpower_en22:00:330
22976997323,4cyclictest18032-21apt22:20:170
22976997320,4cyclictest7850irq/56-dwc_otg21:35:170
2297699731,18cyclictest27709-21ssh22:31:130
22976997116,44cyclictest25943-21idleruntime20:50:390
22976997020,28cyclictest27887-21latency_hist23:15:170
2297699692,22cyclictest0-21swapper/022:09:000
2297699691,56cyclictest0-21swapper/022:56:510
22976996914,8cyclictest8050irq/56-dwc_otg_21:55:180
2297699682,22cyclictest0-21swapper/021:21:300
2297699673,13cyclictest0-21swapper/000:31:430
22976996717,10cyclictest0-21swapper/000:14:500
22976996715,3cyclictest0-21swapper/022:42:180
2297699671,20cyclictest32197-21kernelversion21:55:380
2297699671,20cyclictest32197-21kernelversion21:55:380
22976996711,11cyclictest0-21swapper/022:25:500
2297699661,26cyclictest0-21swapper/000:25:540
2297699661,14cyclictest7356-21latency_hist20:00:170
22976996611,12cyclictest0-21swapper/022:11:510
2297699654,13cyclictest18149-21ssh21:35:390
2297699652,23cyclictest0-21swapper/023:38:420
2297699652,16cyclictest0-21swapper/023:55:470
2297699652,16cyclictest0-21swapper/023:55:470
22976996514,4cyclictest0-21swapper/021:25:250
22976996511,13cyclictest32678-21diskmemload23:22:220
22976996417,3cyclictest0-21swapper/019:39:270
2297699641,14cyclictest18757-21ssh22:20:380
2297699641,11cyclictest0-21swapper/023:25:400
2297699633,10cyclictest0-21swapper/023:40:290
2297699632,21cyclictest0-21swapper/000:09:170
2297699632,18cyclictest0-21swapper/021:13:100
2297699631,25cyclictest0-21swapper/020:30:260
2297699631,16cyclictest0-21swapper/020:20:310
2297699621,18cyclictest9687-21wc22:50:390
2297699621,17cyclictest0-21swapper/021:43:410
2297699621,12cyclictest0-21swapper/022:35:340
2297699621,11cyclictest0-21swapper/023:45:300
2297699617,23cyclictest0-21swapper/020:10:320
2297699616,14cyclictest0-21swapper/023:54:370
2297699611,44cyclictest30168-21grep19:30:500
22976996113,11cyclictest0-21swapper/021:46:310
2297699611,22cyclictest24781-21ssh00:38:180
2297699611,22cyclictest24781-21ssh00:38:180
2297699611,14cyclictest7282-21/usr/sbin/munin23:30:440
2297699604,19cyclictest0-21swapper/023:15:490
2297699602,17cyclictest0-21swapper/019:50:360
2297699601,46cyclictest22043-21aten_rbpower_en20:40:360
2297699601,18cyclictest0-21swapper/000:02:230
2297699601,16cyclictest0-21swapper/019:16:210
2297699596,38cyclictest0-21swapper/021:05:350
2297699591,45cyclictest0-21swapper/019:30:170
2297699591,32cyclictest7821-21fschecks_time20:00:340
254892560,3sleep325103-21kworker/3:020:50:183
22976995610,18cyclictest0-21swapper/020:45:310
200552550,2sleep20-21swapper/222:21:352
2297699541,40cyclictest0-21swapper/020:55:380
2297699512,18cyclictest0-21swapper/020:06:310
2297699501,23cyclictest0-21swapper/020:17:250
2297699501,23cyclictest0-21swapper/020:17:250
180872500,1sleep10-21swapper/122:20:181
2297699491,13cyclictest0-21swapper/021:00:320
2297699491,13cyclictest0-21swapper/021:00:320
22977994820,5cyclictest19939-21idleruntime-cro20:35:171
22978994623,5cyclictest32252-21sed21:05:502
2297799467,17cyclictest8050irq/56-dwc_otg_20:10:341
226002463,35sleep00-21swapper/019:08:550
2297899453,7cyclictest8050irq/56-dwc_otg_00:35:342
2297899453,7cyclictest8050irq/56-dwc_otg_00:35:342
2297799457,26cyclictest0-21swapper/100:15:231
22977994519,5cyclictest3444-21/usr/sbin/munin23:25:501
22978994414,17cyclictest0-21swapper/223:50:322
2297799448,23cyclictest0-21swapper/100:25:331
74502420,2sleep10-21swapper/122:05:401
55512420,3sleep13651-21kworker/1:219:55:171
2297999425,12cyclictest7850irq/56-dwc_otg23:35:303
2297899427,18cyclictest8050irq/56-dwc_otg_00:05:182
2297899422,5cyclictest8050irq/56-dwc_otg_23:25:382
22978994217,5cyclictest12443-21latency_hist00:20:182
2297799425,6cyclictest0-21swapper/121:35:261
2297799424,3cyclictest0-21swapper/100:10:361
22977994220,5cyclictest1111-21runrttasks21:23:331
161342420,3sleep231-21ksoftirqd/221:33:282
76112410,3sleep1221ktimersoftd/123:30:351
2297999414,7cyclictest7950irq/56-dwc_otg_23:30:303
2297899417,31cyclictest0-21swapper/219:25:322
2297899417,24cyclictest0-21swapper/222:26:552
2297899416,3cyclictest8050irq/56-dwc_otg_21:55:282
2297899416,3cyclictest8050irq/56-dwc_otg_21:55:282
2297899416,32cyclictest8050irq/56-dwc_otg_22:30:232
22977994125,7cyclictest0-21swapper/119:35:291
22977994123,3cyclictest28953-21cron21:00:171
106642410,2sleep337-21rcuc/300:16:393
2297999407,20cyclictest0-21swapper/320:25:323
22977994018,7cyclictest28143-21fschecks_time21:50:321
300552390,2sleep3381ktimersoftd/321:52:383
2297899391,27cyclictest23208-21rm23:08:342
2297799398,25cyclictest0-21swapper/122:30:231
2297799397,23cyclictest0-21swapper/121:05:231
2297799394,5cyclictest7950irq/56-dwc_otg_21:15:171
2297799391,25cyclictest0-21swapper/123:20:301
2297899387,14cyclictest8050irq/56-dwc_otg_20:10:332
2297899384,30cyclictest18818-21ssh23:45:302
22978993812,7cyclictest26747-21ssh23:55:362
22978993812,7cyclictest26747-21ssh23:55:362
2297899381,24cyclictest29656-21missed_timers19:30:432
2297799388,16cyclictest8050irq/56-dwc_otg_22:40:461
2297799387,22cyclictest0-21swapper/119:25:321
2297799386,21cyclictest22657-21ssh23:50:321
2297799386,21cyclictest0-21swapper/122:50:331
22979993721,4cyclictest5529-21timerwakeupswit21:15:493
22979993714,5cyclictest0-21swapper/322:30:363
2297999371,15cyclictest1284350ssh00:20:413
2297899377,5cyclictest7950irq/56-dwc_otg_21:00:172
2297899376,27cyclictest307-21taskset19:40:332
2297899375,22cyclictest14993-21ssh23:40:322
22978993718,5cyclictest9204-21sh23:32:002
22978993716,5cyclictest5181-21sh22:02:252
22978993710,14cyclictest0-21swapper/223:00:402
2297799379,20cyclictest4301-21unixbench-2d22:00:481
2297799378,13cyclictest8050irq/56-dwc_otg_22:45:541
2297799374,6cyclictest7941-21ssh21:20:181
22977993713,14cyclictest30573-21ssh00:00:371
2297799371,22cyclictest25796-21ls00:40:181
2297799371,22cyclictest25796-21ls00:40:181
22977993712,14cyclictest26281-1kworker/u9:023:14:181
2297999365,15cyclictest8050irq/56-dwc_otg_23:05:373
2297999363,3cyclictest0-21swapper/319:55:293
22979993612,4cyclictest17168-21users00:25:503
22978993612,5cyclictest32678-21diskmemload00:20:322
22978993611,7cyclictest8050irq/56-dwc_otg_22:10:422
2297799368,19cyclictest9316-1kworker/u9:223:42:261
2297799364,14cyclictest8050irq/56-dwc_otg_20:00:471
2297799362,5cyclictest0-21swapper/123:00:321
22977993610,6cyclictest7950irq/56-dwc_otg_21:40:321
2297999357,13cyclictest8050irq/56-dwc_otg_19:40:183
2297999354,10cyclictest0-21swapper/322:00:303
2297899358,7cyclictest7950irq/56-dwc_otg_21:40:322
2297899357,25cyclictest0-21swapper/223:20:322
2297899354,5cyclictest5010-21latency_hist22:45:172
2297899353,19cyclictest0-21swapper/219:10:202
2297899353,19cyclictest0-21swapper/219:10:202
2297899352,20cyclictest5458-21ssh21:15:492
22978993513,10cyclictest1111-21runrttasks20:42:262
2297799358,15cyclictest0-21swapper/119:40:331
2297799356,4cyclictest0-21swapper/100:30:291
22977993528,5cyclictest28770-21ls23:15:401
2297799351,7cyclictest1165-21cat22:40:181
22977993516,4cyclictest6164-21if_wlan019:55:371
2297799351,27cyclictest22315-21latency_hist23:50:181
2297799351,15cyclictest13149-21ssh23:37:121
2297999349,11cyclictest8050irq/56-dwc_otg_19:10:473
2297999349,11cyclictest8050irq/56-dwc_otg_19:10:473
2297999347,5cyclictest8050irq/56-dwc_otg_19:30:353
2297999343,3cyclictest0-21swapper/322:55:293
22979993427,5cyclictest21781-21rm22:25:043
22979993422,5cyclictest20228-21munin-plugin-st23:05:173
2297999341,3cyclictest0-21swapper/320:20:293
22979993412,11cyclictest7639-21netstat22:05:433
2297899347,24cyclictest0-21swapper/219:55:312
2297899345,14cyclictest8050irq/56-dwc_otg_22:57:402
22978993413,10cyclictest31334-21ssh21:55:182
22978993412,5cyclictest7950irq/56-dwc_otg_21:10:452
22978993411,10cyclictest301ktimersoftd/200:05:332
2297799349,13cyclictest0-21swapper/121:25:351
2297799345,19cyclictest25018-21ssh21:45:231
2297999337,12cyclictest8050irq/56-dwc_otg_22:35:483
2297999336,6cyclictest7950irq/56-dwc_otg_22:40:423
2297999336,6cyclictest39-21ksoftirqd/322:25:393
2297999334,5cyclictest23439-21ssh23:50:433
22979993315,5cyclictest5594-21fschecks_time00:10:403
2297999331,25cyclictest31645-21ls21:05:423
2297999331,21cyclictest0-21swapper/320:15:273
2297999331,21cyclictest0-21swapper/320:15:273
2297899338,5cyclictest7950irq/56-dwc_otg_20:35:492
2297899338,5cyclictest7950irq/56-dwc_otg_00:10:362
2297899338,10cyclictest7950irq/56-dwc_otg_20:00:352
2297899334,16cyclictest1140-21snmpd20:30:262
22978993326,5cyclictest30788-21idleruntime-cro21:05:172
22978993326,5cyclictest30788-21idleruntime-cro21:05:172
22978993326,5cyclictest14320-21ssh21:30:112
22978993315,10cyclictest1111-21runrttasks21:39:422
22978993313,9cyclictest31280-21/usr/sbin/munin19:35:382
2297899331,26cyclictest31827-21ls23:20:182
22978993312,10cyclictest143832chrt20:18:412
22978993312,10cyclictest143832chrt20:18:412
2297799338,14cyclictest8050irq/56-dwc_otg_21:05:171
2297799338,14cyclictest8050irq/56-dwc_otg_21:05:171
2297799337,5cyclictest7950irq/56-dwc_otg_20:40:501
2297799335,4cyclictest0-21swapper/120:50:491
2297799331,7cyclictest26846-21fschecks_time23:55:371
2297799331,7cyclictest26846-21fschecks_time23:55:371
2297799331,5cyclictest0-21swapper/122:00:161
2297799331,5cyclictest0-21swapper/122:00:161
2297799331,19cyclictest8050irq/56-dwc_otg_20:35:501
2297799331,19cyclictest24026-21cut20:45:351
22977993310,10cyclictest11115-21fschecks_time22:10:361
15722331,28sleep30-21swapper/321:10:443
116862330,2sleep20-21swapper/223:35:392
2297999329,4cyclictest7850irq/56-dwc_otg22:45:473
2297999328,5cyclictest0-21swapper/322:53:233
2297999327,5cyclictest7850irq/56-dwc_otg00:35:373
2297999327,5cyclictest7850irq/56-dwc_otg00:35:373
2297999327,15cyclictest0-21swapper/323:20:323
2297999325,13cyclictest8050irq/56-dwc_otg_22:15:323
2297999323,14cyclictest8050irq/56-dwc_otg_19:45:423
22979993215,5cyclictest1111-21runrttasks21:27:373
22979993214,5cyclictest1111-21runrttasks21:33:493
2297999321,26cyclictest0-21swapper/321:00:313
2297999321,26cyclictest0-21swapper/321:00:313
2297999321,22cyclictest4888-21latency_hist00:10:173
2297999321,21cyclictest28965-21latency_hist21:00:183
22979993210,4cyclictest37-21rcuc/321:40:393
22979993210,4cyclictest0-21swapper/323:40:473
2297899326,5cyclictest30762-21kworker/u8:000:30:172
2297899324,6cyclictest8050irq/56-dwc_otg_21:23:382
2297899321,6cyclictest7785-21kworker/u8:322:40:182
22978993216,10cyclictest24085-21grep20:45:352
22978993215,10cyclictest5647-21fschecks_count22:45:372
2297799325,5cyclictest7950irq/56-dwc_otg_20:15:281
2297799325,5cyclictest7950irq/56-dwc_otg_20:15:281
22977993214,6cyclictest15166-21if_eth020:20:371
22977993214,11cyclictest178382chrt20:26:411
22977993211,9cyclictest7850irq/56-dwc_otg22:55:441
22977993211,10cyclictest221ktimersoftd/119:35:181
227832326,15sleep222908-21cut19:10:192
2297999319,8cyclictest0-21swapper/321:55:423
2297999319,8cyclictest0-21swapper/321:55:423
2297999319,5cyclictest39-21ksoftirqd/300:00:433
2297999318,4cyclictest0-21swapper/323:10:333
2297999313,13cyclictest7950irq/56-dwc_otg_23:15:393
22979993118,9cyclictest0-21swapper/323:45:383
22979993115,5cyclictest356050wget23:25:343
22979993110,7cyclictest0-21swapper/321:24:073
2297899318,5cyclictest9550irq/48-DMA10:48:092
2297799316,14cyclictest7850irq/56-dwc_otg20:06:301
2297799315,5cyclictest2252-21grep19:45:341
2297799313,23cyclictest8050irq/56-dwc_otg_19:10:201
2297799313,23cyclictest8050irq/56-dwc_otg_19:10:201
2297799311,3cyclictest0-21swapper/122:22:531
197122310,1sleep30-21swapper/321:38:003
2297999307,4cyclictest0-21swapper/322:10:463
2297999307,20cyclictest0-21swapper/319:20:273
2297999305,14cyclictest8050irq/56-dwc_otg_23:55:443
2297999305,14cyclictest8050irq/56-dwc_otg_23:55:443
2297999304,6cyclictest7950irq/56-dwc_otg_00:33:363
2297999302,19cyclictest0-21swapper/320:51:163
2297999301,18cyclictest9696-21ls20:05:323
2297999301,14cyclictest8050irq/56-dwc_otg_19:25:473
2297899308,4cyclictest16331-21apt20:25:172
2297899307,5cyclictest7950irq/56-dwc_otg_19:45:362
2297899307,5cyclictest31-21ksoftirqd/222:05:472
2297899306,4cyclictest0-21swapper/222:15:312
2297899305,5cyclictest7950irq/56-dwc_otg_00:30:322
2297899303,6cyclictest0-21swapper/219:20:172
2297899303,17cyclictest0-21swapper/220:05:272
2297899301,3cyclictest7850irq/56-dwc_otg22:53:432
2297799305,15cyclictest0-21swapper/123:06:531
2297799302,20cyclictest0-21swapper/121:32:401
2297799301,5cyclictest25201-21modprobe19:20:171
2297799301,5cyclictest0-21swapper/119:20:251
2297999293,13cyclictest8050irq/56-dwc_otg_19:50:343
2297999291,13cyclictest0-21swapper/320:13:543
2297899298,6cyclictest7950irq/56-dwc_otg_20:50:372
22978992920,6cyclictest5495-21munin-plugin-st19:55:172
2297999283,5cyclictest24663-21apt-get21:45:233
2297999282,8cyclictest19469-21grep20:30:433
2297999277,17cyclictest0-21swapper/320:40:353
2297999274,11cyclictest7950irq/56-dwc_otg_20:03:553
2297999272,15cyclictest0-21swapper/319:44:033
226282274,15sleep30-21swapper/319:09:193
224322273,14sleep10-21swapper/119:06:541
22979992611,6cyclictest20301-21cut20:35:373
2297999252,14cyclictest0-21swapper/319:18:153
2297899241,7cyclictest25432-21/usr/sbin/munin19:20:362
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional