You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 12:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 11, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
231908799971,93cyclictest0-21swapper/020:25:010
2319087998510,35cyclictest2377666-21sed23:00:020
231908799745,33cyclictest0-21swapper/022:00:020
2319087997420,25cyclictest2327634-21ssh21:15:000
2319097997225,11cyclictest3467602-21systemd-journal21:21:253
2319093997230,12cyclictest2396484-21systemctl23:39:592
2319087997217,14cyclictest2340183-21ssh21:41:090
231909199715,54cyclictest2389433-21ssh23:25:021
2319087997133,33cyclictest364-21snmpd22:06:190
2319087997019,19cyclictest275-21dbus-daemon22:15:010
2319097996924,6cyclictest2346787-21latency_hist21:55:013
2319087996732,30cyclictest2416393-21sshd00:21:250
2319087996732,30cyclictest16850irq/134-800000022:15:140
2319087996732,30cyclictest16850irq/134-800000022:15:140
2319087996716,15cyclictest2386911-21munin-run23:20:000
2319087996711,38cyclictest2334539-21ssh21:29:290
2319097996630,9cyclictest2346972-21sshd21:55:263
231909199662,60cyclictest2356179-21ssh22:15:021
2319097996522,12cyclictest37-21ksoftirqd/323:50:433
231909399655,41cyclictest2391182-21kworker/u8:023:45:022
2319093996512,11cyclictest2377396-21ssh23:00:002
2319093996512,11cyclictest2377396-21ssh22:59:592
2319087996525,11cyclictest2375655-21ssh22:55:580
231908799652,36cyclictest0-21swapper/021:20:590
2319087996518,15cyclictest2408756-21sh00:05:230
2319087996514,14cyclictest2364475-21ssh22:32:260
2319097996432,27cyclictest2386902-21sh23:20:003
2319097996424,10cyclictest37-21ksoftirqd/321:44:323
2319093996441,7cyclictest2372717-21kworker/2:223:15:202
2319097996322,10cyclictest37-21ksoftirqd/322:41:543
2319087996316,44cyclictest0-21swapper/022:39:240
2319087996316,44cyclictest0-21swapper/022:39:230
2319087996314,13cyclictest2378031-21ssh23:00:570
2319087996313,15cyclictest2360197-21sshd22:23:250
2319097996222,9cyclictest37-21ksoftirqd/320:25:023
2319087996221,11cyclictest2396520-21munin-run23:40:000
2319097996125,10cyclictest37-21ksoftirqd/321:34:543
2319097996123,7cyclictest2325245-21date21:05:033
2319087996120,36cyclictest2401433-21latency_hist23:50:030
2319097996025,8cyclictest37-21ksoftirqd/300:11:493
2319097996025,8cyclictest37-21ksoftirqd/300:11:483
2319087996020,36cyclictest2413082-21ssh00:14:570
2319087996020,36cyclictest2413082-21ssh00:14:560
2319087996015,19cyclictest0-21swapper/000:18:510
2319087996011,33cyclictest2424652-21ssh00:39:110
2319097995925,9cyclictest37-21ksoftirqd/300:27:383
2319097995921,12cyclictest37-21ksoftirqd/321:37:123
2319087995924,17cyclictest2337122-21ssh21:35:010
2319087995917,37cyclictest48-21kcompactd023:40:230
2319087995825,30cyclictest12-21ksoftirqd/023:57:270
2319087995816,11cyclictest2421552-21ssh00:32:250
2319087995812,16cyclictest2320777-21latency_hist19:40:010
2319097995725,8cyclictest37-21ksoftirqd/322:07:233
2319097995723,6cyclictest2325294-21diskmemload21:30:033
2319087995728,9cyclictest12-21ksoftirqd/022:55:000
2319097995624,9cyclictest37-21ksoftirqd/319:40:003
231909399567,35cyclictest291rcuc/222:40:032
231909399567,35cyclictest291rcuc/222:40:022
2319087995633,8cyclictest2383221-21ssh23:12:010
2319087995628,23cyclictest2408353-21cron00:05:000
2319087995625,15cyclictest2370928-21ssh22:45:520
2319087995622,30cyclictest12-21ksoftirqd/000:29:100
2319087995618,34cyclictest0-21swapper/023:30:260
2319087995615,14cyclictest2403155-21ssh23:53:530
231909399559,7cyclictest131rcu_preempt21:00:012
2319093995547,5cyclictest2344512-21kworker/2:100:05:002
2319093995546,7cyclictest30-21ksoftirqd/221:20:012
2319087995531,20cyclictest2344657-21ssh21:50:220
2319087995527,12cyclictest2343513-21ssh21:48:180
2319087995518,32cyclictest2319564-21cron19:20:000
2319097995425,8cyclictest37-21ksoftirqd/323:20:053
231909399544,46cyclictest2332596-21ssh21:25:022
2319087995436,14cyclictest2337757-21sshd21:35:570
2319087995427,4cyclictest2388991-21ssh23:24:190
2319087995417,33cyclictest0-21swapper/022:05:000
231908799541,47cyclictest2325300-21munin-run21:10:000
2319097995336,13cyclictest2405951-21ogrotate00:00:003
2319093995344,6cyclictest2344512-21kworker/2:121:55:122
2319093995344,6cyclictest2307170-21kworker/2:219:20:032
2319093995337,12cyclictest1-21systemd20:35:012
2319093995336,13cyclictest1-21systemd21:15:012
231908799538,39cyclictest0-21swapper/020:20:010
2319087995336,12cyclictest2381012-21ssh23:07:240
2319087995333,16cyclictest2319856-21munin-run19:25:010
2319087995326,12cyclictest2361890-21ssh22:26:490
2319087995310,11cyclictest2328944-21ssh21:17:330
2319093995243,5cyclictest2344512-21kworker/2:100:07:552
2319091995233,14cyclictest1-21systemd21:10:021
2319087995235,13cyclictest2322414-21sshd20:14:240
2319087995222,26cyclictest3467602-21systemd-journal22:44:230
2319093995143,5cyclictest2420279-21kworker/2:300:35:502
2319093995142,6cyclictest2344512-21kworker/2:122:00:122
2319093995133,14cyclictest1-21systemd22:20:012
2319093995133,14cyclictest1-21systemd22:20:012
231909399512,12cyclictest364-21snmpd22:51:422
2319091995136,10cyclictest1-21systemd00:10:011
2319091995132,14cyclictest1-21systemd23:15:011
2319087995144,4cyclictest2322192-21munin-run20:10:000
231909399506,5cyclictest131rcu_preempt21:40:012
2319093995044,3cyclictest2372717-21kworker/2:223:50:462
2319093995042,5cyclictest2413166-21kworker/2:200:18:452
2319093995042,5cyclictest2372717-21kworker/2:223:34:022
2319093995042,5cyclictest2344512-21kworker/2:122:45:002
2319093995042,5cyclictest2324804-21kworker/2:322:34:252
2319093995042,4cyclictest2372717-21kworker/2:223:01:122
2319093995041,6cyclictest2324804-21kworker/2:321:34:002
2319087995038,9cyclictest2320091-21sshd19:29:240
2319087995037,10cyclictest2324483-21sshd20:51:260
2319097994928,10cyclictest2356319-21wc22:15:033
231909399496,8cyclictest30-21ksoftirqd/200:23:312
2319093994941,5cyclictest2372717-21kworker/2:223:40:382
2319093994941,5cyclictest2372717-21kworker/2:223:08:392
2319093994941,5cyclictest2344512-21kworker/2:122:26:092
2319093994941,5cyclictest2344512-21kworker/2:122:10:322
2319093994940,6cyclictest2372717-21kworker/2:223:30:012
2319091994938,9cyclictest23-21ksoftirqd/120:10:021
2319091994935,11cyclictest2338757-21sshd21:38:241
231909199492,26cyclictest131rcu_preempt00:20:001
231909199491,42cyclictest3467602-21systemd-journal20:25:001
2319087994918,25cyclictest2321986-21cat20:05:020
2319087994918,25cyclictest2321986-21cat20:05:020
2319097994835,9cyclictest1-21systemd19:35:013
2319097994835,9cyclictest1-21systemd19:35:003
2319097994831,13cyclictest275-21dbus-daemon22:20:013
2319097994831,13cyclictest275-21dbus-daemon22:20:003
2319093994841,4cyclictest2324804-21kworker/2:321:47:422
2319093994840,5cyclictest2372717-21kworker/2:223:10:592
2319093994840,5cyclictest2344512-21kworker/2:123:56:252
2319091994831,13cyclictest23-21ksoftirqd/121:55:311
2319091994822,6cyclictest23-21ksoftirqd/123:02:001
2319087994830,14cyclictest364-21snmpd20:30:310
2319087994826,17cyclictest2324763-21sh21:00:010
2319087994822,21cyclictest2321634-21cron19:59:590
2319087994816,7cyclictest12-21ksoftirqd/019:14:040
2319087994812,10cyclictest2389836-21ssh23:25:370
2319093994740,4cyclictest2344512-21kworker/2:123:24:172
2319093994739,5cyclictest2344512-21kworker/2:122:06:202
2319093994739,5cyclictest2307170-21kworker/2:219:25:082
2319093994738,5cyclictest2344512-21kworker/2:100:14:272
2319093994738,5cyclictest2344512-21kworker/2:100:14:272
2319091994722,16cyclictest275-21dbus-daemon22:30:001
2319087994739,4cyclictest2321145-21munin-run19:50:000
2319087994739,4cyclictest2321145-21munin-run19:50:000
2319087994735,9cyclictest2325006-21sshd21:00:260
2319087994733,10cyclictest2323236-21munin-run20:30:020
231909799464,4cyclictest131rcu_preempt23:05:023
231909799463,6cyclictest131rcu_preempt23:10:013
2319097994632,11cyclictest275-21dbus-daemon22:40:013
2319097994632,11cyclictest275-21dbus-daemon22:40:003
2319093994639,3cyclictest2344512-21kworker/2:100:31:232
2319093994638,5cyclictest2324804-21kworker/2:321:41:322
2319093994638,5cyclictest2323437-21kworker/2:021:05:012
2319093994612,6cyclictest131rcu_preempt21:51:592
2319091994633,7cyclictest2358535-21ssh22:20:001
2319091994633,7cyclictest2358535-21ssh22:19:591
2319087994638,4cyclictest2320905-21munin-run19:45:010
2319087994618,22cyclictest1-21systemd19:35:020
2319087994618,22cyclictest1-21systemd19:35:020
2319097994529,11cyclictest2425098-21systemd-run00:40:003
231909799452,4cyclictest131rcu_preempt21:20:013
231909399459,6cyclictest131rcu_preempt20:24:592
2319093994537,5cyclictest2313028-21kworker/2:320:16:302
2319093994536,6cyclictest2313028-21kworker/2:320:05:422
2319093994530,11cyclictest275-21dbus-daemon20:15:012
2319091994529,12cyclictest2386949-21systemd-run23:20:001
2319091994525,15cyclictest3467602-21systemd-journal19:30:011
2319087994535,6cyclictest2323707-21sshd20:38:250
2319087994512,11cyclictest2324126-21latency_hist20:45:030
2319097994428,12cyclictest1-21systemd19:30:013
2319097994427,13cyclictest275-21dbus-daemon23:40:013
231909799442,4cyclictest131rcu_preempt20:00:013
231909799441,3cyclictest131rcu_preempt22:32:383
231909399445,10cyclictest30-21ksoftirqd/220:42:372
2319093994436,5cyclictest2344512-21kworker/2:122:46:472
2319093994435,5cyclictest2344512-21kworker/2:100:27:272
2319091994428,13cyclictest23-21ksoftirqd/123:30:211
2319087994436,4cyclictest2324260-21munin-run20:50:000
2319097994327,11cyclictest309-21in:imuxsock00:05:013
2319093994335,5cyclictest2323437-21kworker/2:021:09:302
2319093994334,6cyclictest2313028-21kworker/2:319:55:012
231909199439,20cyclictest2327808-21ssh21:15:011
2319091994330,10cyclictest23-21ksoftirqd/121:46:011
231909799427,5cyclictest131rcu_preempt22:28:443
2319097994231,8cyclictest2375420-21sshd22:55:253
2319097994231,8cyclictest2375420-21sshd22:55:253
2319097994223,14cyclictest275-21dbus-daemon22:25:013
2319093994234,5cyclictest2344512-21kworker/2:122:22:512
2319093994234,5cyclictest2323437-21kworker/2:020:35:192
231909199427,6cyclictest131rcu_preempt21:45:001
231909199423,6cyclictest131rcu_preempt19:20:011
2319091994230,10cyclictest23-21ksoftirqd/120:45:021
2319091994226,12cyclictest23-21ksoftirqd/122:07:581
2319091994225,12cyclictest2406017-21systemd-run00:00:011
2319087994234,5cyclictest2321364-21sshd19:50:340
2319097994130,7cyclictest3467602-21systemd-journal19:14:253
2319097994129,8cyclictest2325321-21pluginstate21:10:013
2319097994127,11cyclictest37-21ksoftirqd/322:53:363
2319097994125,12cyclictest37-21ksoftirqd/323:32:103
231909799411,3cyclictest131rcu_preempt21:47:213
2319093994133,5cyclictest2323437-21kworker/2:020:55:012
2319093994128,10cyclictest30-21ksoftirqd/219:40:022
2319091994131,7cyclictest2423125-21sshd00:35:351
2319091994130,8cyclictest23-21ksoftirqd/100:26:501
2319091994127,9cyclictest309-21in:imuxsock21:27:241
2319091994127,10cyclictest23-21ksoftirqd/122:44:061
2319091994127,10cyclictest0-21swapper/122:25:021
2319091994126,12cyclictest23-21ksoftirqd/123:38:061
2319091994126,12cyclictest23-21ksoftirqd/122:56:151
2319091994126,12cyclictest23-21ksoftirqd/122:56:151
2319091994125,12cyclictest23-21ksoftirqd/121:31:541
2319091994124,12cyclictest23-21ksoftirqd/122:46:211
231909799406,3cyclictest150-21pr/ttyS200:21:573
231909799405,3cyclictest131rcu_preempt19:50:023
231909799405,3cyclictest131rcu_preempt19:50:013
231909799403,5cyclictest131rcu_preempt21:11:433
2319097994032,5cyclictest2324006-21latency_hist20:45:013
2319097994026,11cyclictest37-21ksoftirqd/323:43:303
2319097994025,12cyclictest37-21ksoftirqd/322:01:443
2319097994025,11cyclictest37-21ksoftirqd/323:26:493
231909799401,3cyclictest131rcu_preempt23:15:023
2319097994013,6cyclictest131rcu_preempt23:45:583
2319093994028,9cyclictest1-21systemd19:15:002
2319093994026,11cyclictest30-21ksoftirqd/221:29:272
2319091994030,6cyclictest309-21in:imuxsock00:35:011
2319091994026,11cyclictest23-21ksoftirqd/100:11:211
2319091994026,11cyclictest23-21ksoftirqd/100:11:201
2319091994025,4cyclictest23-21ksoftirqd/121:18:161
2319091994022,13cyclictest275-21dbus-daemon22:35:011
231909199401,5cyclictest131rcu_preempt20:55:011
2319091994013,6cyclictest131rcu_preempt23:05:161
2319091994012,5cyclictest131rcu_preempt23:51:421
2319091994010,6cyclictest131rcu_preempt23:50:001
231909799395,4cyclictest131rcu_preempt20:05:033
231909799395,4cyclictest131rcu_preempt20:05:023
2319097993925,7cyclictest2322432-21grep20:15:003
231909799391,3cyclictest131rcu_preempt00:07:433
2319093993924,11cyclictest30-21ksoftirqd/219:35:012
2319093993924,11cyclictest30-21ksoftirqd/219:35:002
2319093993922,12cyclictest3467602-21systemd-journal20:00:012
2319091993930,6cyclictest1-21systemd00:25:021
2319091993929,6cyclictest309-21in:imuxsock19:35:301
2319091993925,11cyclictest23-21ksoftirqd/122:54:191
2319091993925,11cyclictest23-21ksoftirqd/120:00:011
2319091993923,13cyclictest23-21ksoftirqd/123:28:401
2319087993921,15cyclictest2319244-21ls19:10:020
231909799382,5cyclictest0-21swapper/300:30:033
2319097993822,12cyclictest275-21dbus-daemon20:10:013
2319097993820,13cyclictest275-21dbus-daemon20:30:003
2319097993819,13cyclictest275-21dbus-daemon20:40:003
231909799381,4cyclictest131rcu_preempt20:20:013
231909799381,3cyclictest131rcu_preempt22:45:453
2319097993811,6cyclictest131rcu_preempt00:15:373
231909399384,5cyclictest131rcu_preempt19:25:012
2319091993829,7cyclictest23-21ksoftirqd/122:00:061
2319091993826,9cyclictest23-21ksoftirqd/122:39:091
2319091993826,9cyclictest23-21ksoftirqd/122:39:081
2319091993826,9cyclictest23-21ksoftirqd/119:55:001
2319091993825,10cyclictest2319287-21sshd19:11:241
2319091993824,11cyclictest23-21ksoftirqd/100:02:111
2319091993724,10cyclictest23-21ksoftirqd/120:20:001
2319091993723,11cyclictest275-21dbus-daemon19:25:001
2319091993723,11cyclictest1-21systemd20:30:001
2319091993722,12cyclictest23-21ksoftirqd/123:44:291
2319091993720,12cyclictest275-21dbus-daemon21:00:011
231909799363,5cyclictest131rcu_preempt20:55:253
2319097993625,7cyclictest2321046-21sed19:45:023
2319097993623,10cyclictest37-21ksoftirqd/320:55:003
2319097993619,13cyclictest2321430-21cat19:55:013
231909199369,6cyclictest131rcu_preempt21:25:001
2319091993627,6cyclictest2320347-21cron19:35:011
2319091993627,6cyclictest2320347-21cron19:35:001
231909799351,3cyclictest131rcu_preempt19:20:003
2319093993524,9cyclictest30-21ksoftirqd/220:50:002
2319091993526,6cyclictest23-21ksoftirqd/120:37:461
2319091993524,9cyclictest23-21ksoftirqd/121:05:001
2319097993424,7cyclictest37-21ksoftirqd/319:25:023
2319097993423,8cyclictest2323490-21pluginstate20:35:023
2319093993422,9cyclictest30-21ksoftirqd/219:47:502
2319093993422,9cyclictest30-21ksoftirqd/219:47:492
231909199346,5cyclictest131rcu_preempt19:10:011
2319091993422,9cyclictest23-21ksoftirqd/121:50:551
2319091993422,8cyclictest2324429-21cut20:50:021
2319091993421,10cyclictest23-21ksoftirqd/119:45:011
231909799332,3cyclictest131rcu_preempt20:50:013
2319093993322,8cyclictest30-21ksoftirqd/219:43:112
2319091993323,6cyclictest2323473-21latency_hist20:35:011
2319093993222,8cyclictest30-21ksoftirqd/220:05:022
2319093993222,8cyclictest30-21ksoftirqd/220:05:022
2319093993221,9cyclictest30-21ksoftirqd/220:30:022
2319091993224,4cyclictest23-21ksoftirqd/119:50:021
2319091993224,4cyclictest23-21ksoftirqd/119:50:011
2319091993221,8cyclictest23-21ksoftirqd/120:05:021
2319091993221,8cyclictest23-21ksoftirqd/120:05:021
2319091993022,5cyclictest2322418-21cron20:14:591
231909799281,3cyclictest131rcu_preempt19:10:023
231909399281,4cyclictest131rcu_preempt19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional