You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 10:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Jul 19, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29023256913,540sleep029091-21/usr/sbin/munin21:19:060
20187236923,325sleep220227-21fschecks_count00:18:572
16876234319,305sleep016867-21apt-get18:58:510
5767233437,105sleep30-21swapper/321:49:023
2975923138,291sleep129770-21ntpq19:38:591
16015230917,277sleep016082-21awk00:04:210
18908229946,234sleep30-21swapper/320:48:363
10932229331,198sleep311079-21latency_hist20:23:353
11552228628,245sleep311586-21idleruntime-cro22:08:333
16700227917,211sleep316702-21cron00:08:323
2593622668,219sleep326017-21switchtime21:09:043
792226122,223sleep2855-21/usr/sbin/munin19:49:182
25547225512,227sleep325550-21cron22:53:323
31159223947,108sleep331223-21cat19:43:583
31159223947,108sleep331223-21cat19:43:583
29044223820,20sleep30-21swapper/323:03:503
27365223040,60sleep30-21swapper/322:58:453
3499221111,143sleep33522-21munin-run21:43:333
4045220810,56sleep30-21swapper/323:28:343
3085220426,130sleep33177-21latency_hist19:58:353
2162721989,127sleep321661-21proc_pri22:39:063
2162721989,127sleep321661-21proc_pri22:39:063
11042218817,149sleep011173-21grep22:04:100
9975218440,99sleep39979-21cron22:03:323
9975218440,99sleep39979-21cron22:03:323
23181218027,118sleep323191-21ntp_states22:43:593
3593217835,125sleep30-21swapper/319:58:563
7159217626,91sleep30-21swapper/323:38:353
16709217626,51sleep30-21swapper/322:23:553
5368217449,49sleep30-21swapper/320:04:053
30924217183,40sleep30-21swapper/323:08:593
13373217028,90sleep313350-21latency_hist23:58:353
915021680,24sleep19154-21fschecks_time23:43:541
915021680,24sleep19154-21fschecks_time23:43:541
21256216237,110sleep321268-21cron19:13:323
21256216237,110sleep321268-21cron19:13:323
8320216010,105sleep38397-21apt-config21:58:353
2189521607,133sleep30-21swapper/320:57:413
17824216014,131sleep117895-21latency_hist22:28:341
25979215910,109sleep326100-21latency_hist19:28:353
18149215242,61sleep30-21swapper/322:28:523
1229215119,82sleep30-21swapper/321:34:043
19238214721,107sleep119329-21cut20:48:541
21634214610,97sleep321676-21fschecks_count19:13:533
21634214610,97sleep321676-21fschecks_count19:13:533
15982214533,97sleep015499-21apt-get20:38:430
32180213631,67sleep332267-21cat21:29:073
8718213353,20sleep30-21swapper/320:14:063
23925213042,42sleep30-21swapper/321:03:553
11913212419,88sleep211922-21diskstats22:08:482
2630221238,56sleep30-21swapper/319:28:513
14226212323,61sleep30-21swapper/320:33:543
14226212323,61sleep30-21swapper/320:33:543
361212119,81sleep00-21swapper/019:49:070
32652212021,80sleep132171-21apt-get19:48:441
26914212026,79sleep227067-21grep19:29:062
10622212036,40sleep30-21swapper/323:49:023
19697211930,77sleep319706-21forks22:33:533
19697211930,77sleep319706-21forks22:33:533
6542211820,78sleep06581-21cut20:08:540
29731211819,83sleep229744-21modprobe21:23:322
9261211719,79sleep19289-21awk20:18:331
29528211721,76sleep129596-21switchtime23:04:041
26930211610,56sleep30-21swapper/321:13:483
7409211514,79sleep27471-21/usr/sbin/munin21:54:062
12436211513,86sleep212582-21cat20:28:352
615211416,90sleep00-21swapper/023:15:140
4384211223,69sleep14490-21fschecks_time23:28:521
30791211114,81sleep030845-21sed23:08:560
30447211128,72sleep330482-21ntpq21:23:583
757921097,39sleep37636-21if_eth023:38:563
51921098,61sleep3602-21cat19:49:113
24786210916,25sleep20-21swapper/222:49:032
14877210917,63sleep20-21swapper/220:34:092
14877210917,63sleep20-21swapper/220:34:092
6441210815,74sleep26516-21/usr/sbin/munin23:34:062
20088210828,67sleep320091-21apt-get00:18:463
27768210644,18sleep30-21swapper/319:33:483
27661210418,65sleep027742-21grep21:14:050
19759210419,48sleep319877-21latency_hist19:08:363
13430210436,46sleep213447-21uniq18:48:522
13279210348,37sleep30-21swapper/320:29:053
17349210245,35sleep30-21swapper/320:43:353
1778210115,71sleep01779-21/usr/sbin/munin19:53:480
223821005,11sleep30-21swapper/323:21:113
14688210038,44sleep30-21swapper/322:18:343
14688210038,44sleep30-21swapper/322:18:343
3056829910,51sleep330634-21modprobe19:43:333
284772997,80sleep3370-21snmpd21:18:493
2709829926,33sleep30-21swapper/322:58:333
1120129931,31sleep30-21swapper/320:23:523
204042989,76sleep320407-21cron20:53:323
186302985,11sleep30-21swapper/300:13:533
3252329721,34sleep30-21swapper/323:14:003
1720129718,58sleep217357-21tail22:24:072
100122978,78sleep3370-21snmpd23:45:463
100122978,78sleep3370-21snmpd23:45:463
1229229622,34sleep30-21swapper/322:08:573
417929511,60sleep10-21swapper/119:59:061
2909629523,49sleep229239-21latency_hist19:38:352
2309929520,34sleep30-21swapper/319:18:503
2280529518,60sleep222814-21modprobe19:18:322
2280529518,60sleep222814-21modprobe19:18:322
308902949,18sleep00-21swapper/019:43:500
308902949,18sleep00-21swapper/019:43:500
255442946,75sleep00-21swapper/022:53:290
1795629420,59sleep117980-21netstat20:43:571
1009729410,72sleep210129-21cat22:03:352
1009729410,72sleep210129-21cat22:03:352
172642937,50sleep30-21swapper/300:08:573
172642937,50sleep30-21swapper/300:08:573
172052936,16sleep10-21swapper/118:59:001
2378229227,43sleep223922-21sed19:19:092
1821429232,42sleep30-21swapper/319:03:353
1088529213,66sleep210931-21latency_hist20:23:342
42862918,37sleep30-21swapper/323:28:463
2982529120,58sleep029913-21mailstats19:39:020
1492029118,33sleep30-21swapper/322:18:463
227162899,36sleep30-21swapper/320:59:023
161872888,43sleep316235-21unin-run18:58:343
161872888,43sleep316235-21unin-run18:58:343
1558828825,40sleep215672-21fschecks_time00:04:112
77442878,43sleep37763-21munin-run20:13:343
1560428617,52sleep015725-21wc20:38:350
1560428617,52sleep015725-21wc20:38:350
1275928624,40sleep012849-21fschecks_time20:28:540
942828514,58sleep00-21swapper/023:44:020
942828514,58sleep00-21swapper/023:44:020
767128526,28sleep10-21swapper/121:54:091
3248028514,47sleep232047-21/usr/sbin/munin23:13:592
161102856,69sleep00-21swapper/022:22:230
1703128410,60sleep212826-21kworker/2:118:58:572
1204128318,27sleep00-21swapper/020:24:090
78128212,51sleep0791-21/usr/sbin/munin21:33:520
601928220,44sleep16137-21timerwakeupswit21:49:081
2294328212,55sleep222995-21interrupts22:43:542
2293428221,41sleep223083-21unixbench_multi20:59:052
3138628128,27sleep10-21swapper/121:28:361
2747928121,39sleep127567-21tune2fs22:58:551
283028030,29sleep20-21swapper/223:23:562
270122805,12sleep20-21swapper/221:13:512
1824828019,34sleep10-21swapper/122:28:541
1493627920,28sleep10-21swapper/118:53:511
1493627920,28sleep10-21swapper/118:53:511
2050027822,33sleep020537-21grep00:19:030
1867627828,26sleep00-21swapper/000:14:030
700927515,24sleep00-21swapper/021:53:560
1369227521,14sleep30-21swapper/323:58:513
1086227423,22sleep20-21swapper/222:04:082
2799827320,35sleep028083-21ls22:59:060
1024427313,45sleep10-21swapper/122:03:381
1024427313,45sleep10-21swapper/122:03:381
38192727,48sleep30-21swapper/321:43:493
203522726,9sleep20-21swapper/220:52:512
1602027218,34sleep216074-21sort20:38:532
2086827123,29sleep10-21swapper/122:38:321
2086827123,29sleep10-21swapper/122:38:321
1505427117,20sleep20-21swapper/200:03:342
1708327024,26sleep00-21swapper/000:08:540
1708327024,26sleep00-21swapper/000:08:540
1185327015,39sleep012002-21date23:53:350
771226915,39sleep27741-21cron20:13:332
528226917,27sleep00-21swapper/020:04:020
2465926927,22sleep10-21swapper/122:48:581
1996826916,26sleep00-21swapper/019:08:410
1996826916,26sleep00-21swapper/019:08:410
1854726920,27sleep10-21swapper/119:03:571
1623726930,22sleep0310-21in:imuxsock18:58:350
1623726930,22sleep0310-21in:imuxsock18:58:350
750226812,26sleep20-21swapper/223:38:552
667926816,27sleep00-21swapper/023:34:080
3028326821,23sleep00-21swapper/023:08:350
2509326816,35sleep11-21systemd21:08:331
217826811,12sleep20-21swapper/219:53:582
183152687,11sleep00-21swapper/019:03:360
100322687,52sleep20-21swapper/223:46:022
100322687,52sleep20-21swapper/223:46:022
537926719,30sleep25446-21df21:48:532
3052326714,35sleep130524-21/usr/sbin/munin23:08:491
2508926721,30sleep125126-21ntp_states19:24:001
1232626722,22sleep10-21swapper/122:08:591
776626621,20sleep10-21swapper/120:13:361
273626618,31sleep22815-21/usr/sbin/munin21:39:032
1878526619,24sleep00-21swapper/022:29:060
872326520,26sleep10-21swapper/123:43:341
466926515,23sleep20-21swapper/220:03:352
2687226511,37sleep10-21swapper/122:55:401
2575326513,31sleep0227-21systemd-journal22:53:370
2156326521,20sleep00-21swapper/022:39:030
2156326521,20sleep00-21swapper/022:39:030
1521626521,21sleep20-21swapper/222:18:582
1347726519,20sleep00-21swapper/022:13:550
1347726519,20sleep00-21swapper/022:13:550
399526418,25sleep10-21swapper/123:28:321
357826414,21sleep10-21swapper/121:43:351
284802647,36sleep10-21swapper/121:18:511
2175826414,28sleep10-21swapper/119:13:551
2175826414,28sleep10-21swapper/119:13:551
1281026412,34sleep012943-21unixbench_singl23:54:070
1214626414,34sleep212153-21/usr/sbin/munin23:53:532
2576026310,36sleep225804-21ntpq21:08:592
214626318,24sleep10-21swapper/119:53:571
1777326316,31sleep217836-21/usr/sbin/munin20:43:542
378626218,21sleep00-21swapper/019:59:010
3243926216,32sleep132478-21ntpq23:13:581
2934226219,27sleep229375-21/usr/sbin/munin23:03:572
2446326217,29sleep024607-21latency_hist19:23:350
2282026226,21sleep00-21swapper/022:43:520
2071126217,18sleep00-21swapper/020:53:500
135326214,22sleep00-21swapper/023:18:580
1334726222,20sleep30-21swapper/318:48:493
364726118,24sleep00-21swapper/023:24:110
3187226115,33sleep231886-21kernelversion21:28:582
3066926110,21sleep20-21swapper/223:08:542
1926826111,31sleep219360-21users19:04:102
1067826116,32sleep110721-21ntpq22:04:021
969226020,24sleep19765-21fw_packets20:18:571
919026019,20sleep00-21swapper/021:59:170
919026019,20sleep00-21swapper/021:59:170
512326016,20sleep10-21swapper/120:03:591
2757026016,25sleep20-21swapper/222:58:562
2538426014,32sleep025377-21apt-get21:08:480
2222626016,28sleep122244-21/usr/sbin/munin20:58:441
2183526014,20sleep20-21swapper/222:39:092
2183526014,20sleep20-21swapper/222:39:092
2046326017,27sleep120495-21/usr/sbin/munin19:09:021
2046326017,27sleep120495-21/usr/sbin/munin19:09:021
162326011,33sleep21629-21ntp_states23:19:042
1608326018,21sleep10-21swapper/100:04:221
964225917,20sleep00-21swapper/020:18:560
2621925913,22sleep00-21swapper/019:28:480
2515125916,22sleep00-21swapper/019:24:040
2196225917,18sleep20-21swapper/220:58:322
1498625915,28sleep215010-21/usr/sbin/munin18:53:522
1498625915,28sleep215010-21/usr/sbin/munin18:53:522
148722599,15sleep00-21swapper/000:03:200
1177125917,28sleep111897-21sed20:24:061
513325819,18sleep00-21swapper/021:48:340
381725814,27sleep13812-21apt-get21:43:471
3233625817,25sleep132540-21dpkg19:48:361
3233625817,25sleep132540-21dpkg19:48:361
291025815,21sleep00-21swapper/021:39:050
2747525813,24sleep10-21swapper/121:14:021
2513625819,19sleep20-21swapper/221:08:342
2494325815,27sleep225007-21cat19:23:572
2142425821,18sleep10-21swapper/122:39:001
2142425821,18sleep10-21swapper/122:39:001
1980125817,25sleep119829-21munin-run00:18:321
1898525813,27sleep219008-21ntp_states00:14:092
18652588,17sleep20-21swapper/221:37:262
1806425815,26sleep218060-21apt-get22:28:472
1762825815,20sleep10-21swapper/100:09:071
1762825815,20sleep10-21swapper/100:09:071
1448225815,21sleep10-21swapper/120:34:001
1448225815,21sleep10-21swapper/120:34:001
1049725817,18sleep20-21swapper/223:48:482
2928625716,23sleep20-21swapper/219:38:432
1496825718,25sleep114909-21latency_hist00:03:331
1494425714,26sleep114939-21apt-get22:18:481
1375825717,24sleep113789-21/usr/sbin/munin22:14:001
1375825717,24sleep113789-21/usr/sbin/munin22:14:001
1184125714,29sleep011853-21cpu22:08:460
89212566,13sleep20-21swapper/220:14:092
79225623,18sleep10-21swapper/121:33:531
2598425625,18sleep20-21swapper/222:53:512
2067425613,17sleep10-21swapper/100:19:081
1961525618,19sleep00-21swapper/022:33:500
1961525618,19sleep00-21swapper/022:33:500
1914925614,25sleep019142-21apt-get20:48:490
492625514,20sleep00-21swapper/023:29:030
240112559,25sleep00-21swapper/021:03:560
2345925516,17sleep10-21swapper/122:44:051
1999825516,23sleep219995-21apt-get19:08:472
1999825516,23sleep219995-21apt-get19:08:472
1464725513,21sleep20-21swapper/222:18:332
1464725513,21sleep20-21swapper/222:18:332
807525411,19sleep00-21swapper/020:13:510
3130525414,24sleep231342-21ntp_states19:44:012
3130525414,24sleep231342-21ntp_states19:44:012
3062625414,24sleep030712-21sed21:24:030
2649225417,16sleep10-21swapper/119:28:571
206772545,12sleep20-21swapper/222:36:022
206772545,12sleep20-21swapper/222:36:022
1460325415,23sleep01-21systemd18:53:340
1399325413,24sleep20-21swapper/220:33:362
738125312,24sleep07375-21apt-get23:38:490
637525317,19sleep10-21swapper/123:34:051
449725314,24sleep24611-21timerwakeupswit21:44:082
3227025314,21sleep00-21swapper/021:29:080
25022537,17sleep10-21swapper/123:23:351
2359925315,17sleep00-21swapper/021:03:350
1706725313,24sleep217076-21df_inode00:08:522
1706725313,24sleep217076-21df_inode00:08:522
1572625316,18sleep10-21swapper/120:38:361
64492528,16sleep00-21swapper/021:51:130
2108725213,21sleep121130-21ntpq20:53:591
1314925214,23sleep10-21swapper/120:29:031
1227625214,22sleep112326-21if_eth023:53:561
1705325110,16sleep00-21swapper/022:24:050
2847925020,15sleep10-21swapper/119:34:061
2547325011,22sleep125565-21perl21:08:531
2342925012,23sleep123468-21memory19:19:001
1755725017,17sleep00-21swapper/020:43:400
3130724910,14sleep20-21swapper/221:28:332
2235724910,16sleep00-21swapper/019:14:060
2235724910,16sleep00-21swapper/019:14:060
1342324917,16sleep10-21swapper/118:48:511
1060724914,18sleep10-21swapper/123:49:001
558424712,21sleep25616-21latency_hist23:33:342
3167524719,15sleep10-21swapper/121:28:521
287472478,27sleep00-21swapper/019:34:580
490824312,16sleep20-21swapper/220:03:472
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional