You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 14:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Nov 29, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3810266216,628sleep33886-21/usr/sbin/munin20:38:483
21431252816,494sleep221434-21sshd23:16:162
24902252710,502sleep024939-21ntpq20:03:430
18753243320,393sleep318767-21cron21:28:153
18753243320,393sleep318767-21cron21:28:153
1501240510,67sleep10-21swapper/122:13:401
18949240225,340sleep119084-21awk19:43:491
32462230417,269sleep21-21systemd23:53:152
32462230417,269sleep21-21systemd23:53:152
2355229910,275sleep02363-21awk23:58:420
30838228020,243sleep030935-21fschecks_time20:23:380
26241226313,201sleep11-21systemd23:33:151
8717225227,204sleep18792-21switchtime00:18:471
18003225010,228sleep35666-21snmpd23:05:383
2605522408,194sleep126070-21grep20:08:331
9450223617,25sleep10-21swapper/120:58:161
9450223617,25sleep10-21swapper/120:58:161
2071222966,61sleep10-21swapper/123:58:351
6352222023,149sleep16444-21latency_hist20:48:191
6352222023,149sleep16444-21latency_hist20:48:191
18708221921,177sleep318239-21apt-get23:08:253
24690221831,173sleep124693-21cron23:28:141
24690221831,173sleep124693-21cron23:28:141
7707220657,90sleep17806-21fschecks_time22:33:351
31137219111,164sleep131173-21ntpq20:23:431
3259121898,167sleep132649-21latency_hist23:53:181
3259121898,167sleep132649-21latency_hist23:53:181
8593218313,150sleep08627-21proc_pri20:53:430
8593218313,150sleep08627-21proc_pri20:53:430
23633217947,60sleep123780-21grep19:58:471
12566217216,57sleep10-21swapper/121:08:181
2405216924,124sleep00-21swapper/020:34:450
1104421618,93sleep111115-21ls19:18:451
20414215413,101sleep120476-21fschecks_time23:13:361
9652215311,125sleep19799-21unixbench_singl19:13:481
30739214812,118sleep130416-21/usr/sbin/munin22:03:591
29449214711,90sleep129468-21kernelversion20:18:381
29449214711,90sleep129468-21kernelversion20:18:381
14781214111,75sleep10-21swapper/121:13:511
14781214111,75sleep10-21swapper/121:13:511
13606214110,115sleep113693-21wc22:53:171
11175213957,14sleep10-21swapper/122:43:521
11175213957,14sleep10-21swapper/122:43:521
6640213756,26sleep10-21swapper/120:48:351
3913213565,55sleep13977-21ps00:03:441
10228213550,33sleep10-21swapper/120:58:491
12580213327,40sleep10-21swapper/100:33:161
13252213239,52sleep10-21swapper/121:08:471
20865212919,74sleep10-21swapper/119:50:351
25274212812,96sleep225328-21cpuspeed21:48:302
966821279,84sleep19687-21apt-get00:23:271
21758212712,73sleep10-21swapper/119:53:381
18732212611,76sleep118729-21apt-get23:08:271
17126212617,93sleep017173-21idleruntime23:03:370
1702721266,47sleep10-21swapper/123:03:341
1729212522,40sleep10-21swapper/120:34:251
5617212426,36sleep10-21swapper/119:03:391
5617212426,36sleep10-21swapper/119:03:391
17890212416,69sleep117933-21ntpq21:23:411
17890212416,69sleep117933-21ntpq21:23:411
16694212121,38sleep10-21swapper/119:38:281
459621199,51sleep14482-21latency_hist20:43:181
4051211933,69sleep14207-21apt-config22:23:171
325211916,74sleep10-21swapper/123:53:301
1106521190,13sleep20-21swapper/221:03:172
9474211818,82sleep39616-21latency_hist00:23:173
27642211635,33sleep127647-21cpu20:13:331
30411211512,87sleep329925-21apt-get22:03:253
28619211334,64sleep128638-21apt-get21:58:261
28619211334,64sleep128638-21apt-get21:58:261
17347211215,79sleep317389-21latency_hist21:23:183
17347211215,79sleep317389-21latency_hist21:23:183
28329211112,82sleep128506-21cat21:58:181
29408211019,75sleep129422-21latency_hist23:43:171
2000211016,79sleep32064-21latency20:34:363
6353210813,72sleep00-21swapper/000:13:160
6353210813,72sleep00-21swapper/000:13:160
5795210839,24sleep10-21swapper/122:28:181
9781210721,67sleep39847-21fschecks_time20:58:393
30967210727,26sleep10-21swapper/123:48:171
4770210616,73sleep24915-21date00:08:172
22269210527,42sleep122297-21cut21:38:431
22269210527,42sleep122297-21cut21:38:431
2204821058,61sleep10-21swapper/123:18:371
2204821058,61sleep10-21swapper/123:18:371
15672210523,70sleep115751-21cat22:58:381
8216210427,56sleep38229-21fschecks_count20:53:343
8216210427,56sleep38229-21fschecks_count20:53:343
7082210424,65sleep27154-21mailstats00:13:472
3203021047,60sleep132069-21df22:08:321
29423210313,70sleep20-21swapper/223:43:182
16480210316,73sleep316607-21latency_hist19:38:183
13411210226,38sleep113541-21latency_hist19:28:181
3906210122,66sleep33997-21fschecks_time18:58:363
6272210020,55sleep26330-21sendmail-msp00:13:142
6272210020,55sleep26330-21sendmail-msp00:13:142
3891210054,23sleep00-21swapper/020:38:490
15845210022,28sleep20-21swapper/219:33:442
1990129917,56sleep219916-21fschecks_count19:48:342
1349629916,70sleep00-21swapper/022:52:520
2554129620,25sleep10-21swapper/121:48:371
677729510,12sleep00-21swapper/000:13:370
3237629515,66sleep232456-21tune2fs20:28:372
2225029514,65sleep322260-21df_inode21:38:413
2225029514,65sleep322260-21df_inode21:38:413
2218629516,57sleep222148-21/usr/sbin/munin21:38:402
2218629516,57sleep222148-21/usr/sbin/munin21:38:402
3204629415,63sleep232086-21sed20:28:172
139542947,49sleep113968-21grep19:28:381
2032129315,25sleep00-21swapper/019:48:450
630429254,24sleep16341-21grep22:28:381
630429254,24sleep16341-21grep22:28:381
587229216,58sleep05910-21grep19:03:460
587229216,58sleep05910-21grep19:03:460
671029114,60sleep36733-21latency_hist19:08:173
671029114,60sleep36733-21latency_hist19:08:173
1905429119,13sleep10-21swapper/121:28:381
89412908,17sleep30-21swapper/319:13:303
671529019,50sleep26847-21grep22:28:472
671529019,50sleep26847-21grep22:28:472
3237129019,12sleep10-21swapper/120:28:361
3208729013,61sleep332233-21latency_hist20:28:183
2633528919,24sleep30-21swapper/323:33:173
2158928921,49sleep3225-21systemd-journal23:18:143
77232888,20sleep17759-21ntpq19:08:391
77232888,20sleep17759-21ntpq19:08:391
2111228715,55sleep10-21swapper/121:33:511
2111228715,55sleep10-21swapper/121:33:511
1514228716,21sleep00-21swapper/021:13:590
1514228716,21sleep00-21swapper/021:13:590
143122878,68sleep114342-21ps22:53:451
280842860,9sleep00-21swapper/020:13:440
2715228614,53sleep027238-21threads23:33:460
56492857,67sleep35666-21snmpd22:27:033
944228415,49sleep09536-21/usr/sbin/munin22:38:430
307192846,69sleep00-21swapper/023:45:460
2819028416,52sleep328195-21/usr/sbin/munin23:38:343
2579728322,18sleep10-21swapper/120:08:161
159372838,34sleep10-21swapper/121:18:281
159372838,34sleep10-21swapper/121:18:281
65532829,57sleep30-21swapper/320:48:293
1556728218,17sleep00-21swapper/019:33:380
301728135,28sleep22934-21latency_hist20:38:182
2575928123,42sleep05666-21snmpd21:48:440
1642928138,30sleep10-21swapper/119:38:161
2474128011,56sleep324802-21latency_hist23:28:163
2474128011,56sleep324802-21latency_hist23:28:163
2137228014,38sleep021507-21unixbench_singl21:33:540
2137228014,38sleep021507-21unixbench_singl21:33:540
1030728023,32sleep20-21swapper/200:23:482
1709627816,32sleep20-21swapper/219:38:412
1095227816,19sleep30-21swapper/319:18:433
875427721,30sleep00-21swapper/019:13:190
762427719,35sleep07627-21sshd20:50:520
3041527725,30sleep20-21swapper/222:03:282
1346727715,46sleep213553-21wc21:08:522
2426727622,33sleep024391-21cat20:03:190
1797827622,33sleep018010-21munin-plugin-st19:43:150
586427522,26sleep20-21swapper/222:28:192
277562755,58sleep10-21swapper/123:37:271
2430127418,25sleep20-21swapper/221:43:492
2409527414,40sleep00-21swapper/021:43:430
2469427322,31sleep224740-21unin-run23:28:152
2469427322,31sleep224740-21unin-run23:28:152
950527216,25sleep20-21swapper/220:58:172
804127211,45sleep38064-21ntp_states22:33:413
242222729,49sleep124300-21/usr/sbin/munin21:43:481
234627228,28sleep32353-21ntp_states23:58:413
209032726,13sleep30-21swapper/321:33:443
209032726,13sleep30-21swapper/321:33:443
762327120,21sleep30-21swapper/319:08:373
762327120,21sleep30-21swapper/319:08:373
3208327129,26sleep00-21swapper/022:08:350
2319127122,24sleep00-21swapper/023:23:160
2319127122,24sleep00-21swapper/023:23:160
2060127114,21sleep30-21swapper/319:48:483
1223427011,45sleep012290-21if_eth019:23:370
1212527016,32sleep20-21swapper/219:23:352
106302706,28sleep10-21swapper/122:43:201
974226916,40sleep39819-21switchtime22:38:503
819526920,22sleep00-21swapper/000:18:330
4582695,15sleep30-21swapper/323:53:363
2815926913,41sleep328242-21gpgv21:58:163
2722126915,31sleep00-21swapper/021:53:410
1893326915,37sleep00-21swapper/023:09:310
1884326925,25sleep20-21swapper/221:28:182
1884326925,25sleep20-21swapper/221:28:182
1586326914,37sleep215878-21/usr/sbin/munin22:58:432
61902686,29sleep10-21swapper/100:12:121
61902686,29sleep10-21swapper/100:12:121
2919026822,22sleep30-21swapper/320:18:323
2919026822,22sleep30-21swapper/320:18:323
2706426822,25sleep30-21swapper/323:33:453
108302687,52sleep00-21swapper/021:01:430
78402675,30sleep10-21swapper/100:18:061
2712126718,23sleep20-21swapper/221:53:392
1887126712,15sleep00-21swapper/019:43:480
1393326714,31sleep00-21swapper/022:53:360
1180326721,21sleep00-21swapper/021:03:450
2631226619,20sleep20-21swapper/223:33:162
2159826615,22sleep20-21swapper/219:53:352
1807826630,16sleep30-21swapper/319:43:183
1238726618,22sleep20-21swapper/200:31:482
803526517,27sleep00-21swapper/019:08:470
508026519,26sleep25116-21munin-run19:03:152
2650726513,37sleep026513-21ntp_states20:08:440
2487126514,32sleep324900-21/usr/sbin/munin20:03:423
2226226510,25sleep00-21swapper/021:38:420
2226226510,25sleep00-21swapper/021:38:420
2184526515,34sleep021851-21ntp_states19:53:410
2062526522,22sleep20-21swapper/221:33:312
2062526522,22sleep20-21swapper/221:33:312
1396926512,20sleep20-21swapper/219:28:392
1197426522,24sleep30-21swapper/322:48:143
1197426522,24sleep30-21swapper/322:48:143
737526412,35sleep07486-21latency_hist22:33:180
737526412,35sleep07486-21latency_hist22:33:180
2800826423,22sleep20-21swapper/220:13:432
157712645,11sleep30-21swapper/322:58:403
1015626413,38sleep310176-21/usr/sbin/munin00:23:453
891926320,26sleep28954-21sed22:38:162
127602639,28sleep20-21swapper/222:48:462
82282626,12sleep20-21swapper/200:18:352
326592625,12sleep30-21swapper/320:28:423
2498226122,22sleep30-21swapper/321:48:143
1728826119,23sleep20-21swapper/221:23:172
1728826119,23sleep20-21swapper/221:23:172
101782616,11sleep00-21swapper/000:23:460
531626015,28sleep35371-21memory00:08:393
531626015,28sleep35371-21memory00:08:393
2871526010,24sleep00-21swapper/021:59:180
2871526010,24sleep00-21swapper/021:59:180
2742426018,19sleep20-21swapper/220:13:182
2313926021,18sleep20-21swapper/223:23:142
2313926021,18sleep20-21swapper/223:23:142
2030326010,26sleep220326-21munin-run21:33:152
1905926017,26sleep019042-21apt-get21:28:410
1873526018,27sleep20-21swapper/219:43:462
1495326013,33sleep215021-21switchtime21:13:562
1495326013,33sleep215021-21switchtime21:13:562
1250026019,27sleep312530-21idleruntime-cro00:33:143
723125916,23sleep20-21swapper/220:48:482
323125917,25sleep00-21swapper/022:18:440
316325911,21sleep30-21swapper/322:18:433
3035625915,24sleep329925-21apt-get22:03:213
3035625915,24sleep329925-21apt-get22:03:213
296252596,13sleep20-21swapper/221:59:362
296252596,13sleep20-21swapper/221:59:362
140125911,20sleep30-21swapper/322:13:383
1260025914,20sleep20-21swapper/221:08:192
116242590,6sleep30-21swapper/321:03:393
62325816,26sleep3624-21sshd22:10:523
549325814,21sleep00-21swapper/020:44:060
549325814,21sleep00-21swapper/020:44:060
3026125821,17sleep20-21swapper/223:43:462
265525811,18sleep20-21swapper/222:18:262
169162584,11sleep20-21swapper/223:03:192
1671425816,20sleep00-21swapper/023:03:170
157362585,12sleep30-21swapper/321:18:183
157362585,12sleep30-21swapper/321:18:183
1370625814,27sleep013715-21/usr/sbin/munin21:08:540
109625810,21sleep00-21swapper/020:33:150
870625718,16sleep20-21swapper/220:53:452
790225721,22sleep30-21swapper/300:18:163
427125712,31sleep24273-21sshd20:40:492
259052574,11sleep30-21swapper/320:08:183
2146825717,24sleep321474-21diskstats19:53:323
1960325718,23sleep319665-21sensors21:28:563
14392576,40sleep00-21swapper/023:56:120
1046525723,19sleep20-21swapper/219:18:262
939625615,19sleep00-21swapper/019:13:430
381525614,24sleep03816-21apt-get18:58:290
353225619,24sleep03605-21fschecks_time00:03:350
344525613,26sleep33440-21apt-get00:03:303
316072565,39sleep00-21swapper/022:06:570
2604225611,25sleep326170-21/usr/sbin/munin21:48:473
2421325618,21sleep00-21swapper/023:23:490
2421325618,21sleep00-21swapper/023:23:490
1873325616,23sleep218729-21apt-get23:08:282
1240825611,31sleep312435-21awk19:23:403
89225513,18sleep20-21swapper/222:13:172
3048525513,25sleep230498-21munin-run20:23:162
3048525513,25sleep230498-21munin-run20:23:162
2316325513,24sleep323220-21irqstats19:58:373
164142559,13sleep00-21swapper/021:18:470
164142559,13sleep00-21swapper/021:18:470
1406525512,17sleep20-21swapper/222:53:382
1284025517,15sleep30-21swapper/322:48:473
1134025516,18sleep00-21swapper/000:28:350
2969325410,22sleep00-21swapper/020:18:450
2969325410,22sleep00-21swapper/020:18:450
2317225414,22sleep323189-21munin-run23:23:153
2317225414,22sleep323189-21munin-run23:23:153
196725417,24sleep21870-21/usr/sbin/munin23:58:322
1493225410,29sleep014959-21cat19:33:180
1081525413,26sleep210851-21fschecks_count22:43:452
1081525413,26sleep210851-21fschecks_count22:43:452
63232535,12sleep20-21swapper/219:03:522
63232535,12sleep20-21swapper/219:03:522
441825317,18sleep10-21swapper/118:58:471
2810825315,19sleep00-21swapper/023:38:310
256612536,35sleep20-21swapper/220:06:562
1429225315,19sleep30-21swapper/322:53:433
1123625310,29sleep011312-21systemctl22:43:550
1123625310,29sleep011312-21systemctl22:43:550
966325216,19sleep29741-21mailstats22:38:492
827125218,15sleep00-21swapper/022:33:460
548525211,26sleep35491-21ntp_states20:44:053
548525211,26sleep35491-21ntp_states20:44:053
265612529,15sleep20-21swapper/223:33:322
2538925215,15sleep00-21swapper/023:28:420
1749025213,22sleep017486-21apt-get21:23:280
1749025213,22sleep017486-21apt-get21:23:280
1383325214,21sleep313862-21irqstats19:28:363
1287725214,23sleep312880-21df21:08:363
2998525111,22sleep330024-21memory23:43:393
2302125115,21sleep223025-21df19:58:322
2032825114,21sleep020322-21apt-get23:13:310
1728525013,17sleep30-21swapper/319:38:463
121925015,16sleep00-21swapper/022:13:350
312802497,15sleep00-21swapper/023:48:330
312802497,15sleep00-21swapper/023:48:330
135124911,21sleep21400-21irqstats22:13:372
1025424913,15sleep20-21swapper/219:18:172
80824817,17sleep20-21swapper/223:53:522
685024813,16sleep30-21swapper/322:28:483
685024813,16sleep30-21swapper/322:28:483
3169624813,16sleep30-21swapper/323:48:443
3169624813,16sleep30-21swapper/323:48:443
700024714,15sleep20-21swapper/219:08:202
700024714,15sleep20-21swapper/219:08:202
503824712,17sleep00-21swapper/022:24:130
2784924710,18sleep30-21swapper/320:13:403
113862468,17sleep00-21swapper/019:18:480
527824310,16sleep20-21swapper/220:44:002
527824310,16sleep20-21swapper/220:44:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional