You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 03:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 24, 2025 00:44:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21937248927,4857sleep320259-21kworker/3:021:06:383
32622288210,858sleep132582-21apt-get23:24:551
24277284614,811sleep322155-21kworker/u8:321:15:153
17534281410,787sleep117543-21/usr/sbin/munin00:19:391
24746238411,358sleep024784-21ntpq22:59:500
24746238411,358sleep024784-21ntpq22:59:500
3083123266,308sleep020808-1kworker/0:1H23:19:490
18640230222,194sleep30-21swapper/319:14:283
10477229825,209sleep310557-21fschecks_time22:14:433
10477229825,209sleep310557-21fschecks_time22:14:433
5834226380,168sleep35855-21fschecks_count21:59:413
22499226216,229sleep022519-21idleruntime-cro22:54:210
12816226173,118sleep3934-21cron20:39:213
12816226173,118sleep3934-21cron20:39:213
12052224813,222sleep112062-21ntpq20:35:191
12052224813,222sleep112062-21ntpq20:35:191
3929224011,216sleep33944-21latency_hist21:54:243
29080223566,83sleep30-21swapper/321:29:443
11218223277,76sleep30-21swapper/320:34:243
11218223277,76sleep30-21swapper/320:34:243
16443222830,176sleep216509-21unin-run19:09:232
17949221396,94sleep30-21swapper/300:19:573
5499221210,183sleep35533-21proc_pri23:39:513
18344221245,96sleep30-21swapper/319:14:243
10094220120,165sleep210100-21modprobe22:14:212
31028220030,156sleep30-21swapper/319:54:223
22919219176,27sleep30-21swapper/321:09:543
22919219176,27sleep30-21swapper/321:09:543
3353218714,94sleep30-21swapper/320:09:243
19914218470,87sleep30-21swapper/322:44:433
1253921808,110sleep312617-21ls22:19:563
1253921808,110sleep312617-21ls22:19:563
27113217744,43sleep327259-21latency_hist21:24:233
27113217744,43sleep327259-21latency_hist21:24:233
22245217434,49sleep316591-21cyclictest00:34:473
22245217434,49sleep316591-21cyclictest00:34:473
31686217275,80sleep331712-21munin-run21:39:213
27812216914,138sleep227814-21sshd19:42:382
28786216814,77sleep30-21swapper/319:44:573
15873216812,60sleep30-21swapper/300:14:363
15873216812,60sleep30-21swapper/300:14:363
10912216744,99sleep310969-21sendmail-msp23:59:213
15719216517,131sleep015845-21unixbench_multi22:29:540
15719216517,131sleep015845-21unixbench_multi22:29:540
5509215818,125sleep25453-21latency_hist21:59:232
32687215870,18sleep30-21swapper/321:39:543
3263821578,94sleep332656-21grep19:59:233
3948215614,127sleep03987-21ntpq20:09:500
3948215614,127sleep03987-21ntpq20:09:500
20656215639,24sleep30-21swapper/321:04:243
4897215321,113sleep30-21swapper/320:14:253
4897215321,113sleep30-21swapper/320:14:253
24226215224,106sleep024216-21apt-get21:14:390
28754214967,16sleep30-21swapper/323:14:233
31407214735,91sleep30-21swapper/323:20:083
23959214331,17sleep30-21swapper/319:29:473
26078214235,20sleep30-21swapper/323:04:423
28513213930,91sleep028602-21/usr/sbin/munin19:44:500
12001213619,24sleep30-21swapper/323:59:543
16371213312,106sleep216517-21cat22:34:242
16371213312,106sleep216517-21cat22:34:242
16590213222,24sleep30-21swapper/320:49:473
14972213239,78sleep315093-21ls00:09:593
14972213239,78sleep315093-21ls00:09:593
14463213232,76sleep114605-21latency_hist20:44:241
10102213250,23sleep31-21systemd22:14:223
18032213114,100sleep318049-21kernelversion20:54:433
14041213018,96sleep114068-21kworker/u8:200:09:211
14041213018,96sleep114068-21kworker/u8:200:09:211
3176212616,96sleep23215-21modprobe23:34:212
13507212512,23sleep20-21swapper/220:39:482
2871421240,9sleep20-21swapper/223:14:212
19463212436,73sleep319524-21latency_hist22:44:233
19463212436,73sleep319524-21latency_hist22:44:233
13871212419,87sleep313476-21/usr/sbin/munin22:24:483
21751212316,90sleep021778-21/usr/sbin/munin19:24:230
321621218,55sleep33227-21munin-run23:34:223
13347211955,48sleep30-21swapper/320:39:453
704621189,97sleep37084-21processes23:44:523
23403211815,72sleep30-21swapper/322:54:533
1504121158,16sleep30-21swapper/322:29:373
1504121158,16sleep30-21swapper/322:29:373
6550211314,16sleep30-21swapper/320:19:263
2098421137,13sleep00-21swapper/021:04:420
9788210811,84sleep29818-21cpu20:29:382
9788210811,84sleep29818-21cpu20:29:382
3711210813,77sleep13745-21/usr/sbin/munin23:34:481
27549210826,16sleep30-21swapper/321:24:443
27549210826,16sleep30-21swapper/321:24:443
22774210711,78sleep222888-21sed00:34:582
22774210711,78sleep222888-21sed00:34:582
2624121066,11sleep30-21swapper/319:37:383
2624121066,11sleep30-21swapper/319:37:383
14432210512,76sleep014461-21latency_hist20:44:230
7511210330,19sleep37563-21cat22:04:453
2423210320,66sleep02564-21grep21:49:240
2211221027,16sleep30-21swapper/319:24:423
30436210113,71sleep030506-21df21:34:380
21246210122,17sleep30-21swapper/322:49:353
30181210011,77sleep130187-21/usr/sbin/munin21:34:231
26963210022,14sleep30-21swapper/319:39:443
863729812,20sleep30-21swapper/320:24:513
799729813,26sleep30-21swapper/320:24:233
239129816,18sleep30-21swapper/321:49:233
2312829812,72sleep223254-21/usr/sbin/munin21:09:572
2312829812,72sleep223254-21/usr/sbin/munin21:09:572
108929812,21sleep30-21swapper/319:59:553
108929812,21sleep30-21swapper/319:59:553
57872970,7sleep20-21swapper/220:15:082
2953729712,61sleep129643-21latency_hist19:49:231
1174229713,68sleep011818-21awk23:59:510
169529617,61sleep11728-21awk20:04:211
169529617,61sleep11728-21awk20:04:211
144129616,58sleep01475-21latency_hist23:29:220
140382967,28sleep30-21swapper/300:09:193
140382967,28sleep30-21swapper/300:09:193
46902957,76sleep2197-21jbd2/mmcblk2p2-20:12:342
46902957,76sleep2197-21jbd2/mmcblk2p2-20:12:342
2106129526,13sleep30-21swapper/300:29:543
2027829522,56sleep020353-21cat22:44:520
1915729522,15sleep30-21swapper/300:24:453
1640829533,43sleep116442-21idleruntime-cro19:09:211
3259529420,13sleep30-21swapper/323:24:373
2049729438,42sleep30-21swapper/319:19:373
1735629413,69sleep217369-21grep19:09:592
2007429222,44sleep120218-21grep20:59:581
1720129211,68sleep317306-21timerandwakeup22:34:543
1720129211,68sleep317306-21timerandwakeup22:34:543
154792928,16sleep30-21swapper/320:44:543
2637429127,41sleep126412-21idleruntime-cro19:39:221
2637429127,41sleep126412-21idleruntime-cro19:39:221
1979129121,44sleep119830-21fschecks_count22:44:411
243392897,14sleep30-21swapper/322:59:373
243392897,14sleep30-21swapper/322:59:373
1075728915,19sleep00-21swapper/022:14:480
1075728915,19sleep00-21swapper/022:14:480
71002889,62sleep07181-21mailstats23:44:530
987328716,55sleep19763-21/usr/sbin/munin20:29:421
987328716,55sleep19763-21/usr/sbin/munin20:29:421
2589928727,27sleep10-21swapper/121:19:411
2589928727,27sleep10-21swapper/121:19:411
62132869,12sleep10-21swapper/121:59:511
3041428612,36sleep330551-21latency_hist23:19:243
1461128627,33sleep20-21swapper/200:09:502
1461128627,33sleep20-21swapper/200:09:502
1100128619,46sleep111130-21sort23:59:231
179232859,24sleep00-21swapper/022:39:230
179232859,24sleep00-21swapper/022:39:230
98632849,51sleep00-21swapper/020:29:400
98632849,51sleep00-21swapper/020:29:400
3077428412,55sleep230282-21apt-get23:19:272
1722528418,48sleep217273-21grep00:19:232
1722528418,48sleep217273-21grep00:19:232
543028318,41sleep05458-21munin-run21:59:210
2772128216,24sleep20-21swapper/221:24:472
2772128216,24sleep20-21swapper/221:24:472
2678428217,27sleep00-21swapper/019:39:410
1784328213,50sleep20-21swapper/222:39:212
1784328213,50sleep20-21swapper/222:39:212
394928112,51sleep04095-21latency_hist21:54:250
3005028113,49sleep00-21swapper/019:49:470
2788328116,43sleep027930-21/usr/sbin/munin21:24:530
2788328116,43sleep027930-21/usr/sbin/munin21:24:530
3266228012,43sleep0349-21latency_hist19:59:240
1255127921,25sleep10-21swapper/100:04:241
860427819,36sleep18740-21latency_hist22:09:241
34692787,13sleep30-21swapper/323:34:383
2947027820,40sleep229535-21grep23:14:502
176427815,19sleep00-21swapper/020:04:250
176427815,19sleep00-21swapper/020:04:250
1760727811,52sleep217632-21fschecks_time00:19:482
531027720,32sleep20-21swapper/223:39:462
2907827715,29sleep00-21swapper/023:14:400
2549127716,44sleep225537-21/usr/sbin/munin19:34:472
2549127716,44sleep225537-21/usr/sbin/munin19:34:472
2497427714,46sleep225102-21unixbench_multi21:15:422
1249827722,34sleep20-21swapper/200:04:212
2358127614,49sleep223602-21cpu19:29:382
2266127618,30sleep00-21swapper/021:09:460
2266127618,30sleep00-21swapper/021:09:460
2038527616,26sleep20-21swapper/200:29:252
328027522,40sleep00-21swapper/020:09:170
171442755,13sleep10-21swapper/120:49:571
1014527515,47sleep010218-21latency_hist22:14:230
3212727414,34sleep132234-21latency_hist23:24:221
2791827411,50sleep027955-21ps23:09:520
2721427430,20sleep00-21swapper/023:09:230
92232735,12sleep30-21swapper/323:52:273
323027219,30sleep10-21swapper/123:34:241
2599527212,42sleep226076-21grep23:04:412
2052427212,42sleep120541-21/usr/sbin/munin00:29:321
3240227114,33sleep00-21swapper/021:39:470
2279027114,30sleep00-21swapper/022:54:370
2048627113,41sleep220605-21users22:44:542
1785727116,19sleep00-21swapper/020:54:400
1287927121,27sleep00-21swapper/000:04:430
1287927121,27sleep00-21swapper/000:04:430
1242827122,27sleep10-21swapper/122:19:541
1242827122,27sleep10-21swapper/122:19:541
80372707,13sleep00-21swapper/020:24:240
3083127022,26sleep10-21swapper/121:34:471
3074627011,45sleep1369-21snmpd19:50:411
2870327017,34sleep128712-21modprobe23:14:211
2180227021,30sleep221832-21munin-run00:34:212
1873827021,21sleep10-21swapper/120:55:381
51826914,38sleep2519-21diskstats19:59:422
51826914,38sleep2519-21diskstats19:59:422
51542696,28sleep30-21swapper/321:55:453
2334426914,30sleep10-21swapper/100:39:211
2334426914,30sleep10-21swapper/100:39:211
1585926919,21sleep30-21swapper/319:05:123
3025526819,28sleep130322-21http23:19:221
2842426812,36sleep228477-21/usr/sbin/munin19:44:482
1852226814,40sleep118551-21ntpq19:14:261
870926714,37sleep28217-21/usr/sbin/munin20:24:542
2687626721,25sleep10-21swapper/119:39:421
2422826720,29sleep10-21swapper/121:14:401
22042677,15sleep10-21swapper/121:47:311
66232665,12sleep10-21swapper/120:19:281
237052665,49sleep10-21swapper/122:54:591
2184626619,24sleep20-21swapper/219:24:252
1236326612,39sleep212405-21ntpq22:19:512
1236326612,39sleep212405-21ntpq22:19:512
728926524,19sleep10-21swapper/120:20:051
666626520,23sleep20-21swapper/223:44:432
2898726513,27sleep20-21swapper/221:29:432
1877026511,27sleep10-21swapper/122:39:521
1877026511,27sleep10-21swapper/122:39:521
1532626510,41sleep015342-21fschecks_time19:05:010
607826417,21sleep00-21swapper/023:41:380
262622649,13sleep00-21swapper/021:19:480
262622649,13sleep00-21swapper/021:19:480
2346626411,38sleep023522-21cut00:39:240
2346626411,38sleep023522-21cut00:39:240
1822926417,32sleep018235-21modprobe19:14:210
893426321,19sleep10-21swapper/122:09:441
1653726319,22sleep10-21swapper/100:15:061
1653726319,22sleep10-21swapper/100:15:061
1475326317,25sleep00-21swapper/022:29:220
146226315,31sleep21493-21/usr/sbin/munin21:44:462
507326223,22sleep10-21swapper/120:14:271
2136926212,35sleep121432-21fschecks_time22:49:421
1855226218,26sleep218639-21sort19:14:272
756626113,21sleep00-21swapper/022:04:460
3196026120,22sleep20-21swapper/219:54:542
3178826113,32sleep231898-21cat21:39:242
3014626115,30sleep030171-21munin-run21:34:220
2403526116,28sleep124041-21ntp_states19:29:491
230842615,17sleep00-21swapper/019:26:410
2539226015,25sleep00-21swapper/019:34:450
2539226015,25sleep00-21swapper/019:34:450
2182626016,17sleep20-21swapper/222:49:552
204112606,43sleep00-21swapper/021:01:510
2030126014,24sleep00-21swapper/000:29:230
1960526017,22sleep00-21swapper/019:14:580
1934526016,22sleep219342-21apt-get20:59:392
1476426014,19sleep20-21swapper/220:44:412
1003126017,20sleep20-21swapper/223:54:482
941625921,20sleep10-21swapper/123:54:231
384425912,31sleep03890-21ntp_kernel_pll_23:34:510
2627825916,27sleep226341-21awk21:19:502
2627825916,27sleep226341-21awk21:19:502
2444925916,18sleep20-21swapper/222:59:442
2444925916,18sleep20-21swapper/222:59:442
236562599,32sleep023665-21/usr/sbin/munin19:29:400
2220925913,31sleep222236-21date21:09:242
1707125923,22sleep10-21swapper/122:34:521
1707125923,22sleep10-21swapper/122:34:521
365325814,25sleep13662-21/usr/sbin/munin20:09:431
68442574,10sleep20-21swapper/222:02:142
415325715,24sleep14172-21apt-get21:54:341
3226325710,32sleep132276-21irqstats21:39:441
2416225713,23sleep20-21swapper/222:59:242
2185425715,22sleep00-21swapper/000:34:230
2049525716,24sleep120490-21apt-get19:19:351
867825613,24sleep08701-21ntp_states20:24:520
809425615,25sleep08113-21apt-get23:49:330
76972566,41sleep20-21swapper/220:21:022
63312567,39sleep00-21swapper/020:18:420
2212825615,20sleep10-21swapper/119:24:441
2204425612,28sleep222111-21df19:24:412
1350225614,25sleep113495-21apt-get22:24:381
2089325513,25sleep120890-21apt-get21:04:341
1746625511,26sleep017533-21df00:19:380
1514725512,25sleep115250-21perl22:29:431
1514725512,25sleep115250-21perl22:29:431
1193425514,22sleep00-21swapper/022:19:320
1193425514,22sleep00-21swapper/022:19:320
1065525511,29sleep210729-21cat22:14:462
1065525511,29sleep210729-21cat22:14:462
3139525415,23sleep131402-21diskstats19:54:401
273125416,17sleep20-21swapper/221:49:462
2382425417,19sleep123880-21munin-run21:14:241
2382425417,19sleep123880-21munin-run21:14:241
1681325412,17sleep00-21swapper/000:15:090
1681325412,17sleep00-21swapper/000:15:090
1884825315,18sleep20-21swapper/222:39:532
1884825315,18sleep20-21swapper/222:39:532
1827625313,18sleep20-21swapper/220:54:502
1775425311,19sleep10-21swapper/119:10:351
152882535,37sleep10-21swapper/100:10:301
152882535,37sleep10-21swapper/100:10:301
1350325315,22sleep213570-21df22:24:392
1329725316,20sleep20-21swapper/200:04:542
1329725316,20sleep20-21swapper/200:04:542
84192525,34sleep20-21swapper/222:07:392
219225212,24sleep22245-21/usr/sbin/munin20:04:432
163062525,12sleep20-21swapper/220:49:422
115842529,15sleep20-21swapper/220:34:392
115842529,15sleep20-21swapper/220:34:392
70092519,16sleep00-21swapper/022:04:220
3200425112,22sleep20-21swapper/221:39:372
255562516,35sleep10-21swapper/123:03:311
255562516,35sleep10-21swapper/123:03:311
172452515,35sleep00-21swapper/020:51:170
1476125112,23sleep114763-21diskstats20:44:401
958225013,26sleep19624-21latency_hist20:29:251
786325012,22sleep17880-21munin-run23:49:221
2735325014,19sleep10-21swapper/121:24:381
2735325014,19sleep10-21swapper/121:24:381
244725014,22sleep10-21swapper/120:04:501
21712509,17sleep00-21swapper/023:30:030
1900525011,23sleep119002-21apt-get00:24:351
272624910,15sleep10-21swapper/121:49:451
966224812,18sleep00-21swapper/023:54:380
836724810,14sleep20-21swapper/223:49:432
472024813,19sleep24746-21munin-run23:39:212
1488924812,18sleep00-21swapper/020:44:430
1211224813,17sleep00-21swapper/020:35:220
1211224813,17sleep00-21swapper/020:35:220
2599024714,18sleep10-21swapper/123:04:401
11322475,31sleep20-21swapper/223:25:532
561424616,17sleep10-21swapper/123:39:531
3016824613,12sleep20-21swapper/219:49:492
104632459,20sleep110469-21diskstats22:14:411
104632459,20sleep110469-21diskstats22:14:411
144132439,19sleep014419-21diskstats00:09:400
144132439,19sleep014419-21diskstats00:09:400
183742407,14sleep00-21swapper/022:39:430
183742407,14sleep00-21swapper/022:39:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional