You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 04:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Jun 27, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1399599260,25cyclictest8908-21ssh22:37:272
1399599260,25cyclictest18389-21ssh22:16:302
1399499261,24cyclictest15255-21sendmail20:20:001
13996992523,1cyclictest0-21swapper/323:37:293
13996992523,1cyclictest0-21swapper/323:16:123
1399699250,24cyclictest7309-21diskmemload23:43:573
1399699250,1cyclictest0-21swapper/321:11:123
13995992522,0cyclictest151rcu_preempt23:53:552
1399599250,24cyclictest0-21swapper/200:30:282
1399599250,24cyclictest0-21swapper/200:27:392
1399399250,24cyclictest0-21swapper/022:00:130
13996992423,1cyclictest0-21swapper/321:32:013
13996992423,1cyclictest0-21swapper/300:34:233
13996992423,0cyclictest0-21swapper/322:09:293
1399699240,23cyclictest7309-21diskmemload22:17:113
1399699240,23cyclictest0-21swapper/321:20:023
1399699240,23cyclictest0-21swapper/321:20:023
1399699240,1cyclictest9955-21diskmemload23:07:153
1399599240,23cyclictest7309-21diskmemload23:56:002
1399599240,23cyclictest7309-21diskmemload00:20:082
1399599240,0cyclictest7309-21diskmemload22:02:022
1399599240,0cyclictest0-21swapper/223:13:282
1399499240,24cyclictest0-21swapper/122:24:461
1399499240,24cyclictest0-21swapper/121:55:161
1399499240,23cyclictest22713-21ssh23:19:071
1399499240,23cyclictest22244-21sh22:50:091
1399499240,23cyclictest0-21swapper/121:44:541
13993992423,1cyclictest0-21swapper/022:48:040
13993992422,1cyclictest0-21swapper/022:34:380
13993992421,2cyclictest16643-21rm21:17:370
13993992421,2cyclictest15556-21rm23:11:440
13993992421,1cyclictest151rcu_preempt21:40:150
1399399241,1cyclictest4538-21apt-get21:05:020
1399399241,0cyclictest7309-21diskmemload23:45:260
1399399240,24cyclictest0-21swapper/021:46:280
1399399240,23cyclictest8150-21apt-get21:10:430
1399399240,23cyclictest0-21swapper/021:30:320
1399399240,23cyclictest0-21swapper/020:25:430
1399399240,0cyclictest7309-21diskmemload22:13:070
1399399240,0cyclictest0-21swapper/023:37:280
1399399240,0cyclictest0-21swapper/022:29:130
13996992323,0cyclictest7309-21diskmemload00:01:033
13996992323,0cyclictest0-21swapper/323:14:143
13996992323,0cyclictest0-21swapper/322:25:153
13996992323,0cyclictest0-21swapper/300:14:343
13996992321,1cyclictest0-21swapper/323:20:573
13996992321,1cyclictest0-21swapper/322:03:163
13996992321,1cyclictest0-21swapper/321:52:233
13996992320,2cyclictest25191-21ssh23:50:413
1399699230,23cyclictest0-21swapper/323:57:393
1399699230,23cyclictest0-21swapper/322:52:023
1399699230,23cyclictest0-21swapper/300:10:003
1399699230,22cyclictest7309-21diskmemload23:00:283
1399699230,22cyclictest302-21apt-get22:30:433
1399699230,22cyclictest28347-21diskmemload00:21:003
1399699230,22cyclictest25433-21sh00:19:443
1399699230,22cyclictest22249-21ssh23:46:503
1399699230,22cyclictest10637-21sh00:35:313
1399699230,0cyclictest7309-21diskmemload21:58:283
1399699230,0cyclictest7309-21diskmemload21:35:353
13995992323,0cyclictest0-21swapper/223:47:532
13995992323,0cyclictest0-21swapper/221:55:562
13995992323,0cyclictest0-21swapper/221:35:192
13995992321,1cyclictest32756-21rm23:27:312
13995992321,1cyclictest0-21swapper/221:45:432
13995992321,1cyclictest0-21swapper/200:05:492
1399599230,2cyclictest9944-21sh23:07:142
1399599230,23cyclictest0-21swapper/223:19:482
1399599230,23cyclictest0-21swapper/222:30:582
1399599230,23cyclictest0-21swapper/221:42:232
1399599230,22cyclictest3258-21idleruntime-cro19:55:012
1399599230,22cyclictest29905-21ssh22:27:012
1399599230,22cyclictest25691-21idleruntime-cro00:20:002
1399599230,22cyclictest24649-21ssh22:51:092
1399599230,22cyclictest21880-21ssh21:21:552
1399599230,22cyclictest21880-21ssh21:21:552
1399599230,22cyclictest19357-21ssh22:46:332
1399599230,22cyclictest13436-21ssh00:37:032
1399599230,22cyclictest0-21swapper/222:55:222
1399599230,22cyclictest0-21swapper/219:45:422
1399599230,22cyclictest0-21swapper/200:01:232
1399599230,0cyclictest0-21swapper/223:04:412
1399599230,0cyclictest0-21swapper/222:07:152
1399599230,0cyclictest0-21swapper/221:31:402
1399599230,0cyclictest0-21swapper/221:19:342
1399599230,0cyclictest0-21swapper/221:14:542
13994992323,0cyclictest7309-21diskmemload23:51:561
13994992323,0cyclictest0-21swapper/123:36:541
13994992323,0cyclictest0-21swapper/122:35:421
13994992323,0cyclictest0-21swapper/121:38:471
13994992323,0cyclictest0-21swapper/121:10:031
13994992322,1cyclictest0-21swapper/123:07:151
13994992321,2cyclictest17577-21ssh00:11:311
13994992321,1cyclictest1754-21ssh21:33:261
13994992321,1cyclictest0-21swapper/121:45:541
1399499230,2cyclictest20374-21tail23:45:441
1399499230,2cyclictest16904-21ssh22:45:301
1399499230,2cyclictest15705-21switchtime23:40:491
1399499230,23cyclictest0-21swapper/123:13:191
1399499230,23cyclictest0-21swapper/122:41:571
1399499230,23cyclictest0-21swapper/122:29:381
1399499230,23cyclictest0-21swapper/121:18:541
1399499230,23cyclictest0-21swapper/100:26:291
1399499230,23cyclictest0-21swapper/100:21:021
1399499230,22cyclictest23581-21perf19:30:001
1399499230,22cyclictest20506-21ssh00:15:151
1399499230,22cyclictest16435-21sh22:15:431
1399499230,1cyclictest313-21systemd-journal21:30:011
1399499230,0cyclictest31716-21ssh22:00:421
13993992323,0cyclictest0-21swapper/023:01:250
13993992323,0cyclictest0-21swapper/022:57:560
13993992323,0cyclictest0-21swapper/022:51:120
13993992323,0cyclictest0-21swapper/021:21:500
13993992323,0cyclictest0-21swapper/021:21:500
13993992322,1cyclictest62550irq/127-eno123:07:190
13993992321,1cyclictest0-21swapper/023:25:570
13993992321,1cyclictest0-21swapper/023:15:090
13993992320,2cyclictest29276-21sh21:29:300
13993992320,2cyclictest12869-21awk20:15:000
1399399230,23cyclictest62550irq/127-eno100:18:200
1399399230,23cyclictest0-21swapper/023:41:220
1399399230,22cyclictest8835-21sh00:33:240
1399399230,22cyclictest5918-21memory22:05:480
1399399230,22cyclictest15302-21ssh00:10:430
1399399230,0cyclictest0-21swapper/023:58:400
1399399230,0cyclictest0-21swapper/023:30:330
1399399230,0cyclictest0-21swapper/000:05:420
13996992222,0cyclictest0-21swapper/322:45:353
1399699222,19cyclictest0-21swapper/320:10:473
13996992221,1cyclictest7309-21diskmemload22:10:393
13996992220,2cyclictest31446-21ssh00:25:023
13996992220,1cyclictest7309-21diskmemload22:35:423
13996992220,1cyclictest0-21swapper/322:20:253
1399699220,22cyclictest0-21swapper/323:25:343
1399699220,21cyclictest15605-21apt-get20:20:423
1399699220,21cyclictest14707-21sh21:45:193
1399699220,21cyclictest0-21swapper/323:31:103
1399699220,21cyclictest0-21swapper/321:41:263
1399699220,21cyclictest0-21swapper/321:15:513
1399699220,1cyclictest0-21swapper/322:42:223
1399699220,1cyclictest0-21swapper/321:26:253
1399699220,0cyclictest0-21swapper/322:56:303
13995992220,1cyclictest0-21swapper/221:50:512
13995992219,2cyclictest7304-21cron21:10:002
13995992219,2cyclictest17200-21rm00:11:042
1399599220,21cyclictest2499-21ssh23:30:122
1399599220,21cyclictest23990-21ssh23:20:332
1399599220,21cyclictest23778-21ssh22:21:112
1399599220,21cyclictest0-21swapper/223:40:412
1399599220,1cyclictest11209-21ssh22:40:222
1399599220,1cyclictest0-21swapper/223:35:022
1399599220,1cyclictest0-21swapper/222:10:222
1399599220,1cyclictest0-21swapper/221:25:292
13994992222,0cyclictest0-21swapper/123:26:141
13994992221,1cyclictest6969-21ssh22:06:141
1399499220,2cyclictest2615-21rm22:31:231
1399499220,22cyclictest4883-21sh23:02:581
1399499220,22cyclictest13576-21sh00:37:141
1399499220,22cyclictest0-21swapper/122:55:161
1399499220,21cyclictest4222-21proc_pri23:30:491
1399499220,21cyclictest25005-21/usr/sbin/munin23:20:461
1399499220,21cyclictest2146-21munin-run21:00:001
1399499220,21cyclictest10474-21latency_hist20:10:001
1399499220,1cyclictest7725-21ssh00:03:251
1399499220,1cyclictest21029-21users21:20:521
1399499220,1cyclictest21029-21users21:20:521
1399499220,1cyclictest1368-21ssh23:57:291
1399499220,1cyclictest12558-21ssh22:11:141
1399499220,1cyclictest12084-21ssh00:06:441
1399499220,1cyclictest0-21swapper/121:05:001
1399499220,0cyclictest0-21swapper/100:33:181
13993992222,0cyclictest7309-21diskmemload22:40:110
13993992222,0cyclictest0-21swapper/023:21:240
13993992221,1cyclictest7309-21diskmemload21:55:440
13993992220,2cyclictest25981-21ssh22:24:170
13993992220,1cyclictest792-21awk19:50:010
1399399220,22cyclictest13969-21sh00:37:510
1399399220,22cyclictest0-21swapper/023:52:010
1399399220,22cyclictest0-21swapper/022:35:490
1399399220,22cyclictest0-21swapper/000:00:280
1399399220,1cyclictest141rcuc/020:50:010
1399399220,1cyclictest0-21swapper/021:37:450
1399399220,1cyclictest0-21swapper/000:24:040
13996992119,2cyclictest2105-21sh21:00:003
13996992119,1cyclictest18761-21cron19:20:013
1399699210,0cyclictest0-21swapper/319:25:443
13995992119,1cyclictest26042-21munin-run19:35:002
1399599210,0cyclictest0-21swapper/220:20:422
13994992120,1cyclictest5244-21apt-get21:05:431
13994992119,1cyclictest11751-21needreboot20:10:471
1399499210,21cyclictest0-21swapper/121:50:401
1399499210,1cyclictest0-21swapper/120:50:421
1399399210,0cyclictest7309-21diskmemload22:15:480
1399399210,0cyclictest0-21swapper/020:00:420
13996992020,0cyclictest0-21swapper/319:20:423
13996992020,0cyclictest0-21swapper/319:10:423
13996992018,1cyclictest22490-21idleruntime-cro20:35:013
1399699200,0cyclictest0-21swapper/321:05:433
1399699200,0cyclictest0-21swapper/319:50:423
13995992020,0cyclictest2455-21apt-get21:00:422
13995992020,0cyclictest0-21swapper/220:55:442
13995992020,0cyclictest0-21swapper/220:30:432
13995992020,0cyclictest0-21swapper/219:55:442
1399599200,0cyclictest0-21swapper/220:05:422
13994992020,0cyclictest0-21swapper/120:20:421
13993992020,0cyclictest0-21swapper/019:50:420
1399399200,0cyclictest0-21swapper/019:35:430
1399399200,0cyclictest0-21swapper/019:15:430
13996991919,0cyclictest0-21swapper/320:50:413
13996991919,0cyclictest0-21swapper/320:40:433
13996991918,1cyclictest0-21swapper/321:00:423
1399699190,0cyclictest0-21swapper/320:45:423
13995991919,0cyclictest28754-21apt-get19:40:432
1399599190,0cyclictest0-21swapper/220:35:432
13994991919,0cyclictest27605-21apt-get20:45:421
13994991919,0cyclictest0-21swapper/119:50:421
13994991919,0cyclictest0-21swapper/119:15:431
13994991918,1cyclictest22795-21chrt20:35:431
13994991918,1cyclictest14286-21apt-get19:10:421
13993991919,0cyclictest29999-21apt-get20:50:410
13993991919,0cyclictest0-21swapper/019:20:420
13993991919,0cyclictest0-21swapper/019:10:420
13996991818,0cyclictest0-21swapper/320:25:423
13996991817,1cyclictest26335-21apt-get19:35:423
13996991816,1cyclictest30846-21dump-pmu-power19:45:003
13994991818,0cyclictest0-21swapper/120:25:431
13994991816,1cyclictest0-21swapper/119:45:001
1399499180,17cyclictest19098-21apt-get19:20:421
13993991818,0cyclictest0-21swapper/020:15:420
13993991818,0cyclictest0-21swapper/019:55:440
13993991817,1cyclictest2117-21turbostat.cron21:00:000
13993991817,1cyclictest0-21swapper/019:40:430
13996991716,0cyclictest0-21swapper/320:05:423
13995991714,2cyclictest12885-21perf20:15:002
13994991715,1cyclictest22486-21perf20:35:011
13993991717,0cyclictest0-21swapper/020:35:430
13993991716,1cyclictest17696-21perf20:25:000
13996991614,1cyclictest817-21basename19:50:013
13996991614,1cyclictest0-21swapper/319:55:453
13995991613,2cyclictest21169-21idleruntime-cro19:25:002
1399599160,15cyclictest26651-21sed20:40:522
1399599160,15cyclictest0-21swapper/220:50:412
13994991616,0cyclictest0-21swapper/120:40:431
13994991614,1cyclictest5521-21seq19:58:341
1399399164,11cyclictest27306-21perf20:45:010
1399399160,15cyclictest0-21swapper/019:30:000
13995991515,0cyclictest0-21swapper/220:25:432
1399499150,14cyclictest0-21swapper/119:45:311
13993991515,0cyclictest0-21swapper/020:30:430
13993991514,1cyclictest23891-21apt-get19:30:420
1399699142,7cyclictest0-21swapper/320:15:423
13996991414,0cyclictest0-21swapper/320:00:063
13996991414,0cyclictest0-21swapper/319:30:313
1399699140,14cyclictest0-21swapper/320:35:053
13995991414,0cyclictest0-21swapper/220:45:282
13995991414,0cyclictest0-21swapper/220:15:362
13995991414,0cyclictest0-21swapper/219:35:032
13995991414,0cyclictest0-21swapper/219:15:572
1399599140,14cyclictest0-21swapper/219:25:482
1399599140,14cyclictest0-21swapper/219:10:182
1399599140,0cyclictest0-21swapper/220:00:152
1399499140,0cyclictest0-21swapper/119:35:021
1399399140,14cyclictest0-21swapper/021:00:120
1399399140,0cyclictest0-21swapper/020:05:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional