You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 10:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu May 21, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11312727118,7241sleep312811-21kworker/3:223:59:113
17274295419,918sleep117284-21/usr/sbin/munin21:24:121
31080266213,632sleep231169-21/usr/sbin/munin20:24:252
26568244924,405sleep326588-21cron21:54:023
26568244924,405sleep326588-21cron21:54:023
1830724426,401sleep0363-21snmpd21:25:450
23491243019,393sleep123259-21latency_hist21:44:071
259224008,380sleep02617-21awk22:19:270
18103239551,25sleep00-21swapper/019:44:050
7674235119,68sleep00-21swapper/000:19:230
12112233511,218sleep012178-21switchtime22:49:310
12112233511,218sleep012178-21switchtime22:49:310
7394233215,277sleep07432-21http20:54:030
27120231832,270sleep027138-21kernelversion21:54:240
20155230848,77sleep00-21swapper/021:34:040
1203722667,219sleep00-21swapper/000:34:130
5532224126,151sleep05563-21idleruntime-cro00:14:020
27315223444,169sleep00-21swapper/020:13:120
20267223123,153sleep020314-21ntpq19:49:260
555422218,202sleep04227-1kworker/0:0H20:45:230
29955220940,110sleep01-21systemd23:49:030
29955220940,110sleep01-21systemd23:49:030
10600220816,135sleep010638-21fschecks_count00:29:210
1238322079,161sleep012442-21latency_hist21:09:050
23023220535,132sleep023066-21grep19:59:190
10134220127,51sleep00-21swapper/022:44:230
10134220127,51sleep00-21swapper/022:44:230
32252219510,133sleep032393-21unixbench_singl22:09:340
32252219510,133sleep032393-21unixbench_singl22:09:340
926221859,108sleep00-21swapper/019:14:250
7155217355,73sleep00-21swapper/022:34:250
10274216644,63sleep00-21swapper/000:29:040
8465216552,75sleep00-21swapper/022:39:180
8222216391,53sleep00-21swapper/020:55:280
22550215210,83sleep022642-21switchtime21:39:340
7234214615,110sleep00-21swapper/019:09:200
19167214438,38sleep00-21swapper/023:14:060
166021429,18sleep00-21swapper/020:34:240
6862214118,99sleep27005-21unixbench_singl20:49:322
3875213844,77sleep03981-21fschecks_time22:24:240
2465621386,12sleep30-21swapper/320:04:233
3488213631,68sleep00-21swapper/020:39:330
30264213531,66sleep00-21swapper/022:04:250
24053213517,95sleep224089-21latency21:44:402
24053213517,95sleep224089-21latency21:44:402
13949213410,108sleep014098-21grep21:14:060
5791213319,94sleep20-21swapper/222:29:302
2144421329,79sleep00-21swapper/019:54:180
15844213243,76sleep015892-21fschecks_count21:19:230
27149213055,52sleep00-21swapper/023:39:180
14370212824,81sleep2225-21systemd-journal22:59:022
14370212824,81sleep2225-21systemd-journal22:59:022
763721270,7sleep30-21swapper/320:54:053
1256021260,11sleep20-21swapper/200:34:302
4370212324,71sleep30-21swapper/300:09:213
2868721226,10sleep00-21swapper/020:15:550
28990212040,35sleep029069-21switchtime21:59:320
20934212015,53sleep00-21swapper/021:34:330
30891211717,86sleep330986-21perl20:24:223
1728521170,8sleep20-21swapper/221:24:122
4788211618,81sleep24846-21irqstats20:44:242
15399211517,77sleep215544-21/usr/sbin/munin22:59:352
8223211418,81sleep38306-21latency_hist22:39:043
11535211330,49sleep211554-21apt-get22:49:132
11535211330,49sleep211554-21apt-get22:49:132
25796211241,57sleep00-21swapper/021:49:310
25796211241,57sleep00-21swapper/021:49:310
19679211234,59sleep00-21swapper/019:49:060
15234211258,18sleep015300-21swap22:59:320
3260211111,86sleep33261-21sshd22:21:123
1721421117,69sleep0363-21snmpd23:05:180
32063211010,65sleep00-21swapper/020:29:030
2443521100,6sleep20-21swapper/220:04:052
1919211021,75sleep31984-21awk20:34:313
21085210922,68sleep221246-21tail21:34:352
17263210915,73sleep317285-21/usr/sbin/munin19:39:323
1165321086,92sleep00-21swapper/019:21:440
23774210618,69sleep123751-21apt-get21:44:271
23774210618,69sleep123751-21apt-get21:44:271
19226210621,67sleep119236-21netstat21:29:331
2197921049,81sleep222047-21df21:39:192
20925210323,47sleep220918-21apt-get23:19:172
9069210012,17sleep30-21swapper/322:39:333
3069121009,76sleep230744-21idleruntime23:49:452
27859210011,76sleep227909-21basename20:14:232
39329920,21sleep30-21swapper/322:14:053
39329920,21sleep30-21swapper/322:14:053
2535729942,40sleep00-21swapper/023:34:050
3191129817,66sleep131955-21ntpq22:09:271
3191129817,66sleep131955-21ntpq22:09:271
1599429818,62sleep316003-21unin-run23:04:043
2101229621,55sleep321078-21fschecks_time23:19:223
1338129640,44sleep013384-21cron19:29:030
2822729521,48sleep327647-21/usr/sbin/munin20:14:323
2222329526,13sleep00-21swapper/023:24:030
2289429441,38sleep017493-21kworker/0:123:24:280
1882129415,23sleep20-21swapper/219:44:312
1058629425,20sleep20-21swapper/222:44:332
1058629425,20sleep20-21swapper/222:44:332
1020429422,47sleep310351-21grep20:59:353
3249429315,58sleep332008-21/usr/sbin/munin23:54:313
1311229322,15sleep00-21swapper/022:54:160
1311229322,15sleep00-21swapper/022:54:160
751129015,60sleep27611-21fschecks_time00:19:212
1806729031,46sleep018137-21cat23:09:260
1472529030,45sleep014798-21ls21:14:340
934428915,61sleep39391-21tr00:24:273
923728917,51sleep18940-21/usr/sbin/munin00:24:251
301122898,17sleep20-21swapper/220:19:362
2662028926,45sleep126701-21latency_hist21:54:041
2662028926,45sleep126701-21latency_hist21:54:041
751028821,53sleep37551-21ps19:09:273
2953528819,55sleep00-21swapper/020:19:260
1391628832,37sleep00-21swapper/019:29:230
591528714,55sleep36004-21fschecks_time00:14:213
2964928714,58sleep2363-21snmpd23:45:092
2964928714,58sleep2363-21snmpd23:45:092
284202870,9sleep10-21swapper/120:14:341
1231828723,43sleep212334-21modprobe21:09:032
864728610,63sleep18709-21cat22:39:231
677928629,38sleep16809-21cpufreq19:09:051
325412869,16sleep00-21swapper/020:29:240
2747528623,40sleep127625-21latency_hist20:14:061
1989328616,37sleep30-21swapper/319:49:173
1252228614,14sleep10-21swapper/119:24:271
551428530,25sleep00-21swapper/019:04:160
3217328531,14sleep00-21swapper/023:54:240
3210928525,32sleep10-21swapper/120:29:041
16032856,67sleep00-21swapper/022:15:150
123352850,8sleep30-21swapper/321:09:033
586828432,38sleep00-21swapper/000:14:180
315782835,67sleep00-21swapper/023:51:570
3063628318,49sleep130687-21fschecks_time23:49:441
2502128330,12sleep00-21swapper/021:49:020
2502128330,12sleep00-21swapper/021:49:020
2485228233,33sleep00-21swapper/020:04:280
814428126,34sleep18178-21munin-run22:39:021
3256828124,39sleep132008-21/usr/sbin/munin23:54:331
315402818,15sleep20-21swapper/222:09:152
315402818,15sleep20-21swapper/222:09:152
2741828131,31sleep30-21swapper/323:39:253
1944528121,46sleep119549-21grep23:14:271
870228019,40sleep18372-21munin-node19:14:101
2915528020,43sleep329189-21ps23:44:293
2915528020,43sleep329189-21ps23:44:293
1278828019,38sleep312901-21grep19:24:333
827427912,29sleep10-21swapper/120:55:291
2279627822,28sleep30-21swapper/319:59:063
1986227822,38sleep20-21swapper/219:49:122
1340427819,27sleep30-21swapper/300:39:053
3014127715,25sleep20-21swapper/222:04:232
296862777,14sleep30-21swapper/320:19:293
2591727716,44sleep226068-21latency_hist20:09:062
375127642,17sleep00-21swapper/000:06:110
1982427615,43sleep219886-21wc23:14:372
1843427618,32sleep10-21swapper/119:44:221
183327620,26sleep20-21swapper/223:59:352
1707427619,28sleep20-21swapper/219:39:262
1343227622,29sleep20-21swapper/219:29:052
1009827615,42sleep310130-21fschecks_count22:44:213
1009827615,42sleep310130-21fschecks_count22:44:213
998127529,11sleep00-21swapper/020:59:320
937227519,40sleep29416-21ntpq19:14:272
57622758,14sleep20-21swapper/219:04:222
2449127518,34sleep323733-21/usr/sbin/munin21:44:503
2449127518,34sleep323733-21/usr/sbin/munin21:44:503
601227412,40sleep36157-21sed22:29:363
402927422,27sleep30-21swapper/300:09:043
3235427415,24sleep30-21swapper/320:29:183
2934827426,28sleep10-21swapper/120:19:231
2655027426,13sleep00-21swapper/023:35:260
2071727419,23sleep30-21swapper/323:19:063
171827423,29sleep11747-21kernelversion20:34:251
2682827325,31sleep126865-21munin-run23:39:021
933027222,27sleep20-21swapper/200:24:262
837427231,23sleep20-21swapper/219:14:072
83327222,30sleep20-21swapper/223:59:022
643327223,24sleep20-21swapper/200:14:332
307727123,26sleep20-21swapper/220:39:232
279727126,24sleep20-21swapper/222:19:332
2702227114,28sleep30-21swapper/320:09:363
243227122,26sleep10-21swapper/122:19:241
558227017,37sleep25665-21wc00:14:042
1338527018,25sleep10-21swapper/122:54:251
1338527018,25sleep10-21swapper/122:54:251
872726924,23sleep10-21swapper/100:24:051
66372699,21sleep06784-21date22:34:050
3244026923,26sleep20-21swapper/220:29:212
2853326914,34sleep228640-21grep21:59:222
257672699,47sleep10-21swapper/120:08:121
2448426817,21sleep20-21swapper/223:29:302
1562626824,20sleep10-21swapper/119:34:291
1543826817,32sleep115499-21modprobe21:19:031
1438026817,22sleep30-21swapper/321:14:253
1085126820,27sleep10-21swapper/119:19:271
957626720,25sleep10-21swapper/120:59:211
888426717,29sleep28916-21/usr/sbin/munin22:39:282
620826712,21sleep10-21swapper/120:49:191
345026717,34sleep33486-21ntp_states20:39:323
2907226719,25sleep10-21swapper/121:59:331
1550726716,34sleep315591-21/usr/sbin/munin19:34:273
2695626611,37sleep227065-21perl21:54:212
247726610,39sleep00-21swapper/000:04:050
1104126622,28sleep211061-21/usr/sbin/munin19:19:322
583026528,19sleep30-21swapper/319:04:233
3039626520,20sleep30-21swapper/322:04:283
2044826510,23sleep10-21swapper/119:49:311
2010826518,23sleep20-21swapper/221:34:022
1856026519,28sleep318578-21apt21:29:033
161502655,49sleep00-21swapper/019:34:480
63126422,20sleep30-21swapper/322:14:173
838226322,21sleep30-21swapper/320:55:313
588726315,26sleep30-21swapper/320:49:053
541526314,20sleep10-21swapper/122:29:211
2751826316,28sleep127563-21/usr/sbin/munin23:39:271
263912637,11sleep20-21swapper/221:51:432
263912637,11sleep20-21swapper/221:51:432
977426216,33sleep29805-21latency20:59:262
2014126218,24sleep30-21swapper/321:34:033
174026214,31sleep10-21swapper/123:59:341
1188126218,19sleep10-21swapper/100:34:061
332626120,18sleep10-21swapper/100:04:341
280826120,23sleep20-21swapper/200:04:232
2163026122,20sleep20-21swapper/219:54:252
1671026117,29sleep116770-21/usr/sbin/munin23:04:301
1530226120,23sleep215196-21/usr/sbin/munin19:34:222
1498026120,28sleep215007-21cat19:34:052
404826020,20sleep30-21swapper/322:24:273
270822609,42sleep20-21swapper/220:10:112
2320026021,22sleep20-21swapper/223:24:342
2310826013,32sleep323110-21diskstats19:59:223
1875426012,23sleep30-21swapper/319:44:283
1615926018,23sleep20-21swapper/221:19:292
761325912,33sleep37663-21if_eth000:19:223
437325911,25sleep10-21swapper/120:44:061
2822825919,18sleep10-21swapper/121:59:051
26012597,11sleep20-21swapper/220:37:412
2373125919,21sleep30-21swapper/323:29:033
1173425923,17sleep30-21swapper/322:49:223
1173425923,17sleep30-21swapper/322:49:223
1106125910,34sleep311136-21expr00:29:323
3031525815,16sleep10-21swapper/122:04:261
2864225819,18sleep30-21swapper/321:59:233
224192587,17sleep10-21swapper/119:55:241
1563525825,18sleep10-21swapper/123:00:021
1392125816,21sleep30-21swapper/321:14:043
1287625818,17sleep10-21swapper/122:54:041
1287625818,17sleep10-21swapper/122:54:041
87725713,28sleep1906-21irqstats22:14:231
2433525722,17sleep10-21swapper/120:04:051
2197325715,25sleep321969-21apt-get21:39:163
1779525717,25sleep317791-21apt-get23:09:173
1761725719,19sleep10-21swapper/123:09:061
1375925713,28sleep113764-21/usr/sbin/munin19:29:201
777925618,21sleep20-21swapper/222:34:352
581925614,25sleep15814-21apt-get00:14:151
398325616,18sleep10-21swapper/122:24:251
2912325616,23sleep129146-21ntp_states23:44:271
2912325616,23sleep129146-21ntp_states23:44:271
1575625615,24sleep315752-21apt-get21:19:173
1147125613,20sleep20-21swapper/200:30:112
836025518,23sleep18372-21munin-node19:14:061
3235225514,24sleep132347-21apt-get20:29:161
3055825515,24sleep330567-21df_inode23:49:223
2058325513,24sleep30-21swapper/321:34:243
1784525515,25sleep217870-21df23:09:202
1439325517,19sleep314449-21munin-run22:59:033
1439325517,19sleep314449-21munin-run22:59:033
890925414,23sleep38950-21ntp_states19:14:123
426625412,25sleep14261-21apt-get00:09:151
2532225414,19sleep30-21swapper/323:34:043
1416525416,22sleep214167-21apt-get21:14:162
2736325320,18sleep30-21swapper/321:54:313
265372536,34sleep20-21swapper/223:35:162
2145225317,23sleep120900-21/usr/sbin/munin23:19:321
2722225214,17sleep20-21swapper/223:39:202
2037425217,18sleep10-21swapper/121:34:141
792925113,22sleep27913-21apt-get20:54:212
47642516,34sleep20-21swapper/222:25:312
2690825116,16sleep30-21swapper/323:39:043
1679925112,23sleep116792-21apt-get19:39:171
1262625110,24sleep212620-21apt-get21:09:192
1050525114,21sleep110508-21apt-get00:29:131
71825016,15sleep20-21swapper/222:14:202
2410825017,15sleep10-21swapper/123:29:211
1500825011,17sleep30-21swapper/319:34:063
1305625011,24sleep113079-21ntp_states21:09:301
1188125017,16sleep30-21swapper/319:24:053
2308524917,15sleep10-21swapper/123:24:331
1631424913,20sleep216320-21df_inode23:04:192
1131324913,22sleep10-21swapper/122:49:041
1131324913,22sleep10-21swapper/122:49:041
458024813,21sleep14648-21df20:44:191
1656424711,20sleep116659-21users21:19:361
1624224711,20sleep316238-21apt-get23:04:163
110612387,16sleep10-21swapper/121:04:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional