You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 22:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Nov 26, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236272603814,6011sleep210873-21kworker/2:209:52:122
14517286013,831sleep214523-21df_inode09:23:412
14517286013,831sleep214523-21df_inode09:23:412
23060256216,526sleep223061-21sendmail-mta08:06:402
12578255131,406sleep1225-21systemd-journal09:18:061
1609724567,399sleep116135-21ntpq07:43:491
2232223567,338sleep122363-21ntpq08:03:521
31058232314,203sleep131210-21latency_hist11:58:251
31058232314,203sleep131210-21latency_hist11:58:251
12901225242,198sleep112906-21cpu09:18:431
23887222537,64sleep10-21swapper/108:08:491
5657222456,128sleep15812-21latency_hist10:38:261
2684322246,205sleep1195-21jbd2/mmcblk2p2-10:03:131
6034222115,192sleep26099-21awk12:18:542
1093219737,115sleep10-21swapper/112:04:071
2754219013,111sleep10-21swapper/112:08:551
7680218919,65sleep10-21swapper/107:18:371
7680218919,65sleep10-21swapper/107:18:371
8407218721,126sleep10-21swapper/112:28:261
7776218414,154sleep27882-21fschecks_time07:18:422
7776218414,154sleep27882-21fschecks_time07:18:422
1077321848,164sleep05666-21snmpd09:10:180
8186217952,24sleep10-21swapper/109:03:371
1209121788,156sleep112125-21date10:58:251
5671217511,125sleep15757-21perl12:18:451
951321738,120sleep10-21swapper/107:24:111
6135217320,138sleep26172-21grep07:13:422
25498216742,72sleep10-21swapper/111:38:551
2478216220,17sleep10-21swapper/110:28:241
22250216150,64sleep10-21swapper/109:48:271
31687216014,131sleep031736-21ntp_kernel_pll_08:33:520
1131621609,139sleep05666-21snmpd09:13:330
30548215326,25sleep10-21swapper/110:13:471
21669215148,88sleep121748-21latency_hist11:28:251
3687215018,114sleep03817-21cat12:13:270
32169214612,21sleep00-21swapper/011:58:570
1859321467,125sleep118742-21date07:53:261
28882214119,105sleep129030-21unixbench_singl11:48:561
31828213910,68sleep10-21swapper/110:18:271
3985213892,21sleep10-21swapper/112:13:361
25352213619,97sleep00-21swapper/008:13:490
4732212811,20sleep10-21swapper/107:08:481
27525212810,60sleep10-21swapper/108:18:591
1110621289,104sleep111049-21latency_hist09:13:261
25175212633,29sleep10-21swapper/108:13:461
2443321269,68sleep10-21swapper/109:53:481
761621246,84sleep10-21swapper/110:43:311
334321248,100sleep00-21swapper/007:03:590
16204212421,77sleep315733-21apt-get09:28:343
3343212318,83sleep33430-21sort10:28:553
307721239,14sleep10-21swapper/107:03:551
24716212120,87sleep324841-21cat09:53:543
3963212020,72sleep00-21swapper/012:13:350
7620211915,29sleep00-21swapper/010:43:340
14697211975,28sleep10-21swapper/111:04:181
865211845,62sleep1902-21ps08:38:531
28780211811,52sleep10-21swapper/110:08:421
27550211816,85sleep227588-21ntp_kernel_pll_10:03:502
2209521188,94sleep122160-21if_eth011:28:451
13977211817,23sleep30-21swapper/309:21:113
664521178,92sleep10-21swapper/108:58:361
24745211516,81sleep224761-21idleruntime-cro11:38:232
1933421159,60sleep10-21swapper/107:53:551
14242211511,15sleep20-21swapper/211:04:072
8398211411,89sleep01-21systemd12:28:250
5572211427,26sleep10-21swapper/108:53:511
24762211414,84sleep024817-21sendmail08:13:240
17049211452,32sleep017187-21grep09:29:530
957421119,52sleep10-21swapper/110:48:521
957421119,52sleep10-21swapper/110:48:521
20108211134,23sleep112-21rcu_sched11:23:241
15160211114,85sleep00-21swapper/007:40:130
1084121108,65sleep110983-21modprobe07:28:271
1084121108,65sleep110983-21modprobe07:28:271
21995210913,78sleep322010-21fschecks_count11:28:433
1822421097,90sleep00-21swapper/007:49:460
15706210912,53sleep115763-21systemd-run09:28:241
15706210912,53sleep115763-21systemd-run09:28:241
1567921098,56sleep10-21swapper/111:08:381
1320521099,63sleep112577-21/usr/sbin/munin07:33:561
9825210815,78sleep39921-21timerandwakeup10:49:023
9825210815,78sleep39921-21timerandwakeup10:49:023
6402210848,49sleep16476-21swap10:38:541
6402210848,49sleep16476-21swap10:38:541
1524521087,68sleep10-21swapper/107:41:191
6746210710,59sleep10-21swapper/107:13:571
7185210619,70sleep37258-21grep12:23:433
31038210619,69sleep031178-21grep10:14:140
5166210425,61sleep35292-21timerwakeupswit07:08:583
13726210427,22sleep113876-21sed11:03:251
13643210419,63sleep213784-21head09:19:122
979721036,63sleep10-21swapper/112:32:011
28582210336,21sleep10-21swapper/108:23:511
28582210336,21sleep10-21swapper/108:23:511
2499210324,58sleep22583-21latency_hist10:28:252
16068210314,71sleep016096-21/usr/sbin/munin07:43:480
30169210018,59sleep20-21swapper/211:53:502
30169210018,59sleep20-21swapper/211:53:502
323772985,61sleep10-21swapper/112:01:241
305792977,48sleep10-21swapper/108:28:561
305792977,48sleep10-21swapper/108:28:561
185192978,51sleep10-21swapper/111:18:241
2163929557,28sleep10-21swapper/108:03:161
2163929557,28sleep10-21swapper/108:03:161
3260929441,18sleep132617-21munin-run08:38:251
3201529423,46sleep332164-21grep11:58:563
3017829413,64sleep330213-21proc_pri11:53:513
3017829413,64sleep330213-21proc_pri11:53:513
2738729415,59sleep027524-21/usr/sbin/munin08:18:580
249402934,12sleep30-21swapper/308:13:273
488429211,18sleep10-21swapper/108:53:271
203829214,22sleep20-21swapper/208:43:432
327429028,41sleep10-21swapper/108:48:261
1054229019,48sleep210629-21latency_hist10:53:252
1054229019,48sleep210629-21latency_hist10:53:252
1725228930,38sleep317253-21cpu07:48:403
894128822,42sleep38955-21apt-get07:23:243
894128822,42sleep38955-21apt-get07:23:243
613728815,56sleep26204-21latency10:38:472
613728815,56sleep26204-21latency10:38:472
2658828816,37sleep30-21swapper/311:43:373
2658828816,37sleep30-21swapper/311:43:373
790828722,41sleep07928-21apt09:03:230
2341428734,41sleep10-21swapper/109:49:241
1926128713,60sleep019272-21ntpq11:18:510
2755728635,41sleep10-21swapper/111:44:401
2755728635,41sleep10-21swapper/111:44:401
1789628615,53sleep317901-21/usr/sbin/munin09:33:433
310428514,51sleep02712-21/usr/sbin/munin10:28:480
2161728525,39sleep321635-21munin-run11:28:233
1642728553,21sleep10-21swapper/109:29:371
244828415,48sleep02477-21cut10:28:230
616928319,46sleep06298-21grep12:18:560
2451428336,14sleep10-21swapper/111:35:241
1234628310,57sleep312409-21df10:58:393
58128212,26sleep20-21swapper/212:03:352
2536428223,28sleep20-21swapper/209:58:252
2484228212,53sleep224938-21latency_hist08:13:262
1607728212,51sleep016113-21ntpq11:08:500
2544228117,48sleep325488-21grep11:38:533
1002928115,49sleep310086-21/usr/sbin/munin09:08:483
1793928026,41sleep118023-21perl09:33:451
1781328023,32sleep30-21swapper/311:13:533
2596327913,53sleep325973-21needreboot09:58:473
2201227913,52sleep022018-21df_inode08:03:430
634327812,51sleep06401-21sendmail_mailqu10:38:530
634327812,51sleep06401-21sendmail_mailqu10:38:530
271492786,14sleep10-21swapper/110:03:371
2009827822,36sleep220122-21modprobe07:58:242
1078427813,49sleep010840-21latency_hist07:28:260
1839527719,27sleep20-21swapper/209:33:582
390127620,35sleep33974-21munin-node07:08:263
660827518,30sleep30-21swapper/307:13:553
573027513,46sleep35856-21timerwakeupswit08:53:543
2128727521,25sleep00-21swapper/009:43:500
658727414,29sleep30-21swapper/310:38:563
658727414,29sleep30-21swapper/310:38:563
3273927421,25sleep00-21swapper/010:19:310
2041927431,31sleep10-21swapper/109:40:481
1964127413,17sleep00-21swapper/009:38:490
263227313,45sleep22637-21ntpq12:08:522
1494427319,37sleep315008-21find09:24:103
1494427319,37sleep315008-21find09:24:103
1287127317,29sleep00-21swapper/010:58:520
419127220,30sleep24323-21unixbench_multi08:48:562
2903327218,23sleep20-21swapper/211:48:582
213632725,17sleep00-21swapper/007:59:470
213632725,17sleep00-21swapper/007:59:470
93927116,32sleep30-21swapper/310:23:263
610127114,38sleep36167-21switchtime12:18:553
319427116,40sleep33199-21cron12:10:233
2012027118,29sleep220253-21latency_hist11:23:252
1411827115,41sleep314125-21dpkg11:03:363
1272127116,27sleep20-21swapper/207:33:452
1236527117,26sleep00-21swapper/007:33:260
1236527117,26sleep00-21swapper/007:33:260
2634927024,28sleep00-21swapper/011:43:260
2256727010,16sleep30-21swapper/309:48:453
432026923,19sleep20-21swapper/212:14:262
386326931,28sleep10-21swapper/110:31:431
2981226921,21sleep00-21swapper/008:28:430
2981226921,21sleep00-21swapper/008:28:430
2788726815,36sleep327936-21unin-run08:23:253
2293826813,28sleep00-21swapper/009:48:540
372026718,23sleep00-21swapper/008:48:460
3092426714,37sleep331035-21sed10:14:133
2065726727,21sleep30-21swapper/307:58:543
2065726727,21sleep30-21swapper/307:58:543
228512667,10sleep00-21swapper/011:29:170
226326622,25sleep22293-21munin-run07:03:242
226326622,25sleep22293-21munin-run07:03:242
1617826614,39sleep216183-21ntp_states07:44:072
1306926613,22sleep30-21swapper/307:33:543
1212626623,18sleep20-21swapper/210:58:262
770526521,25sleep20-21swapper/212:23:552
2870426515,33sleep328771-21/usr/sbin/munin10:08:403
201026523,23sleep10-21swapper/107:00:121
201026523,23sleep10-21swapper/107:00:121
191826515,28sleep30-21swapper/312:08:253
1685626518,28sleep20-21swapper/209:29:502
83332645,12sleep20-21swapper/209:03:462
2332326416,27sleep20-21swapper/211:33:272
2867626315,28sleep228681-21acpi10:08:352
231526323,25sleep31-21systemd07:03:253
231526323,25sleep31-21systemd07:03:253
1599626324,23sleep00-21swapper/009:28:270
1599626324,23sleep00-21swapper/009:28:270
1131826310,14sleep20-21swapper/209:13:352
517526221,19sleep00-21swapper/008:53:420
1752026220,20sleep20-21swapper/207:48:472
54862615,17sleep00-21swapper/007:09:200
532426124,20sleep20-21swapper/208:53:442
2830826113,32sleep028389-21/usr/sbin/munin08:23:460
2830826113,32sleep028389-21/usr/sbin/munin08:23:460
1073426116,28sleep30-21swapper/307:28:253
911026020,25sleep29148-21grep12:28:512
773226019,23sleep30-21swapper/310:43:453
3004826017,26sleep030104-21latency_hist10:13:250
1702126021,20sleep30-21swapper/311:13:253
156092605,11sleep20-21swapper/211:08:272
613325917,26sleep06128-21apt-get07:13:400
2656625913,18sleep20-21swapper/211:43:362
2656625913,18sleep20-21swapper/211:43:362
2585825916,24sleep225944-21users11:39:002
2398525912,32sleep024064-21sendmail_mailtr11:33:530
2258225920,23sleep30-21swapper/308:03:583
864325814,26sleep08638-21apt-get12:28:370
84772589,31sleep08493-21/usr/sbin/munin09:03:480
2671125810,30sleep226602-21/usr/sbin/munin08:18:432
2659025815,26sleep026585-21apt-get11:43:380
2659025815,26sleep026585-21apt-get11:43:380
2079625814,19sleep20-21swapper/207:58:572
2079625814,19sleep20-21swapper/207:58:572
1185225815,19sleep30-21swapper/309:13:563
80025717,19sleep20-21swapper/208:38:502
468625713,29sleep24722-21ntpq10:33:512
2817925714,25sleep328172-21apt-get08:23:413
2817925714,25sleep328172-21apt-get08:23:413
244472575,11sleep30-21swapper/311:34:333
2152125714,22sleep20-21swapper/209:43:562
2091925714,26sleep320912-21apt-get09:43:413
200372575,38sleep00-21swapper/007:57:410
160925718,20sleep00-21swapper/006:58:540
160925718,20sleep00-21swapper/006:58:540
1542925715,19sleep00-21swapper/011:08:250
872325617,15sleep30-21swapper/309:03:553
825425613,23sleep00-21swapper/007:18:530
825425613,23sleep00-21swapper/007:18:530
2844825620,18sleep20-21swapper/208:23:482
2844825620,18sleep20-21swapper/208:23:482
277262560,11sleep30-21swapper/310:03:553
1979825616,23sleep319821-21ntp_states09:38:523
123072568,36sleep20-21swapper/210:58:302
1046125622,19sleep00-21swapper/010:53:230
1046125622,19sleep00-21swapper/010:53:230
431825513,25sleep04311-21apt-get10:33:400
314625517,24sleep23218-21sort07:03:562
281125514,24sleep22822-21diskstats10:28:422
2760825514,24sleep027614-21ntp_states10:03:520
260332555,38sleep20-21swapper/208:14:312
1290025514,24sleep012894-21apt-get09:18:420
117125517,22sleep31173-21apt-get10:23:383
1162725516,24sleep311722-21sed07:28:573
1162725516,24sleep311722-21sed07:28:573
114752559,12sleep20-21swapper/207:28:522
114752559,12sleep20-21swapper/207:28:522
1076625514,24sleep010774-21apt-get10:53:350
1075325516,24sleep30-21swapper/310:53:343
672425415,23sleep26730-21diskstats08:58:412
48425415,23sleep3488-21diskstats08:38:433
3128425411,23sleep331303-21apt-get08:33:383
3033125418,15sleep30-21swapper/308:28:543
3033125418,15sleep30-21swapper/308:28:543
2691425414,23sleep227033-21latency_hist10:03:262
2075225417,18sleep20-21swapper/209:43:272
14652548,35sleep05666-21snmpd08:40:540
1281325410,19sleep00-21swapper/007:33:470
834225315,23sleep20-21swapper/210:44:122
664725312,24sleep36642-21apt-get08:58:383
57525314,20sleep00-21swapper/012:03:330
403525315,22sleep30-21swapper/312:14:063
2811225314,26sleep028121-21acpi11:48:350
2403425312,25sleep024037-21apt-get09:53:350
1809025311,23sleep018183-21memory09:33:480
510125211,21sleep20-21swapper/207:08:572
3252925213,19sleep20-21swapper/212:03:232
2376225213,22sleep323855-21memory08:08:473
2077625217,17sleep00-21swapper/011:23:480
195125215,21sleep01946-21apt-get08:43:370
1771725215,23sleep017723-21ntp_states11:13:500
1415025212,23sleep314146-21apt-get07:38:383
3011525118,19sleep00-21swapper/011:53:480
3011525118,19sleep00-21swapper/011:53:480
2560025113,22sleep025596-21apt-get09:58:360
105072514,16sleep20-21swapper/207:25:332
350425013,21sleep33506-21apt-get08:48:363
1597225011,17sleep30-21swapper/307:43:473
1902224913,21sleep319079-21interrupts11:18:463
1431024917,17sleep20-21swapper/207:38:462
472424812,20sleep34769-21ntp_states10:33:523
3260424815,16sleep20-21swapper/210:19:292
322324811,21sleep33255-21munin-run08:48:243
302592489,20sleep20-21swapper/208:28:532
302592489,20sleep20-21swapper/208:28:532
1928824812,21sleep319289-21ntp_states07:53:533
3154824711,19sleep231572-21munin-run10:18:242
107282474,31sleep20-21swapper/209:09:442
98102468,23sleep39811-21sshd12:32:113
319432445,15sleep20-21swapper/208:33:582
2044524313,17sleep20-21swapper/211:23:422
283882429,17sleep30-21swapper/311:48:463
1731124011,14sleep20-21swapper/211:13:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional