You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Apr 03, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
171797997669,5cyclictest37-21ksoftirqd/320:45:023
17178799751,42cyclictest0-21swapper/000:09:470
171787997332,31cyclictest16850irq/134-800000000:03:580
171797997033,33cyclictest216146-21sshd22:38:023
171797996927,6cyclictest222874-21sh22:53:423
17179799673,19cyclictest182263-21grep21:20:013
17179799672,23cyclictest227772-21unin-run23:05:013
171797996624,12cyclictest3467602-21systemd-journal21:25:023
171787996618,10cyclictest12-21ksoftirqd/021:32:550
171797996514,14cyclictest275-21dbus-daemon19:40:003
171795996516,11cyclictest30-21ksoftirqd/223:25:032
171797996414,9cyclictest284-21systemd-logind23:55:023
171787996423,31cyclictest16850irq/134-800000021:53:230
171797996310,9cyclictest260337-21ssh00:20:023
171787996321,32cyclictest16850irq/134-800000021:42:540
17179799623,18cyclictest3467602-21systemd-journal19:25:003
171797996230,29cyclictest37-21ksoftirqd/323:06:403
171797996230,29cyclictest37-21ksoftirqd/323:06:403
171797996223,8cyclictest37-21ksoftirqd/323:25:253
171797996216,11cyclictest3467602-21systemd-journal21:42:333
171795996215,10cyclictest30-21ksoftirqd/221:25:032
171787996236,17cyclictest23308650ssh23:17:030
171797996125,4cyclictest201002-21sh22:03:073
171797996121,6cyclictest197651-21latency_hist21:55:023
171795996121,9cyclictest131rcu_preempt00:20:062
171795996115,9cyclictest30-21ksoftirqd/222:58:182
171787996139,19cyclictest16950irq/224-800000023:35:010
171797996021,11cyclictest37-21ksoftirqd/322:29:163
171797996019,11cyclictest37-21ksoftirqd/300:00:583
171797996016,11cyclictest37-21ksoftirqd/321:20:243
17179099606,6cyclictest131rcu_preempt00:20:021
171787996019,33cyclictest16850irq/134-800000022:20:090
171787996019,32cyclictest16850irq/134-800000000:22:280
171787996019,31cyclictest16850irq/134-800000023:24:470
171787996014,7cyclictest12-21ksoftirqd/000:35:290
171797995927,5cyclictest218844-21sh22:44:263
171797995927,5cyclictest218844-21sh22:44:253
171797995923,9cyclictest37-21ksoftirqd/323:38:093
171797995923,9cyclictest37-21ksoftirqd/323:38:093
171797995923,9cyclictest37-21ksoftirqd/321:56:103
171797995922,8cyclictest37-21ksoftirqd/323:10:413
171795995925,9cyclictest30-21ksoftirqd/221:34:442
171787995930,14cyclictest238091-21ssh23:28:510
171787995919,31cyclictest16850irq/134-800000022:09:420
171787995919,31cyclictest16850irq/134-800000021:48:280
171787995918,32cyclictest16850irq/134-800000023:57:140
171787995918,32cyclictest16850irq/134-800000023:57:130
171787995918,32cyclictest16850irq/134-800000023:41:290
171787995918,32cyclictest16850irq/134-800000022:53:180
171787995918,32cyclictest16850irq/134-800000022:19:470
171787995918,32cyclictest16850irq/134-800000021:55:160
171787995918,32cyclictest16850irq/134-800000000:30:450
171797995816,8cyclictest309-21in:imuxsock19:45:393
17179599582,9cyclictest30-21ksoftirqd/222:05:012
17179599581,17cyclictest30-21ksoftirqd/200:33:252
17179099585,6cyclictest131rcu_preempt22:25:021
17178799588,45cyclictest0-21swapper/021:00:010
17178799586,18cyclictest249140-21rm23:54:240
171787995833,6cyclictest176435-21latency_hist20:40:020
171787995819,35cyclictest16850irq/134-800000023:06:570
171787995819,35cyclictest16850irq/134-800000023:06:560
171787995817,32cyclictest16850irq/134-800000022:42:120
171787995817,32cyclictest16850irq/134-800000022:42:110
171787995817,32cyclictest16850irq/134-800000000:12:090
171787995815,8cyclictest12-21ksoftirqd/019:20:020
171797995737,14cyclictest3467602-21systemd-journal22:50:013
171797995718,8cyclictest175154-21sort20:15:023
171797995715,12cyclictest284-21systemd-logind19:15:003
17179599577,10cyclictest196774-21sh21:53:192
17179599574,9cyclictest214732-21ssh22:35:002
17179599574,9cyclictest214732-21ssh22:35:002
17179099573,14cyclictest23-21ksoftirqd/100:06:111
17178799574,28cyclictest12-21ksoftirqd/022:37:260
171787995729,12cyclictest264619-21ssh00:30:020
171787995720,6cyclictest16850irq/134-800000022:34:120
171787995720,6cyclictest16850irq/134-800000022:34:110
171787995719,29cyclictest16850irq/134-800000000:16:090
171787995718,32cyclictest16850irq/134-800000022:28:560
171787995718,31cyclictest16850irq/134-800000021:22:360
171787995717,31cyclictest16850irq/134-800000023:46:170
171787995716,32cyclictest16850irq/134-800000021:35:510
171787995711,40cyclictest180538-21ssh21:15:390
171797995616,11cyclictest1-21systemd21:40:003
17179599569,8cyclictest30-21ksoftirqd/221:43:182
17179599565,9cyclictest30-21ksoftirqd/223:15:002
17179599565,6cyclictest30-21ksoftirqd/219:15:012
171790995638,13cyclictest3467602-21systemd-journal23:25:001
171787995618,6cyclictest16850irq/134-800000023:00:430
171787995617,31cyclictest16850irq/134-800000023:39:460
171787995617,31cyclictest16850irq/134-800000023:39:450
171787995617,31cyclictest16850irq/134-800000022:57:280
171787995617,31cyclictest16850irq/134-800000022:12:550
171787995617,31cyclictest16850irq/134-800000022:12:550
171787995616,30cyclictest16850irq/134-800000022:00:070
171797995525,4cyclictest209703-21sh22:23:053
171797995524,8cyclictest37-21ksoftirqd/300:32:263
171797995521,9cyclictest37-21ksoftirqd/300:22:173
17179599559,7cyclictest30-21ksoftirqd/221:10:012
17179599557,7cyclictest30-21ksoftirqd/223:55:042
17179599557,7cyclictest30-21ksoftirqd/223:55:032
17179099551,7cyclictest23-21ksoftirqd/121:25:021
17179099551,5cyclictest131rcu_preempt23:40:011
17179099551,5cyclictest131rcu_preempt23:40:011
171787995517,7cyclictest16850irq/134-800000022:48:580
171787995516,31cyclictest16850irq/134-800000021:29:130
171797995420,8cyclictest0-21swapper/300:00:033
171797995420,8cyclictest0-21swapper/300:00:033
171795995443,8cyclictest291rcuc/200:14:272
171795995411,11cyclictest30-21ksoftirqd/220:45:002
17179099546,6cyclictest131rcu_preempt23:43:231
17178799543,45cyclictest229950-21cron23:10:030
17178799543,18cyclictest172152-21cat19:15:010
17179599536,8cyclictest30-21ksoftirqd/221:20:352
17179599536,5cyclictest30-21ksoftirqd/221:15:032
17179599534,13cyclictest30-21ksoftirqd/222:46:102
171795995335,14cyclictest275-21dbus-daemon22:45:022
171795995335,14cyclictest275-21dbus-daemon22:45:012
17179599532,12cyclictest30-21ksoftirqd/223:15:442
17179099538,8cyclictest23-21ksoftirqd/100:00:011
17179099538,8cyclictest23-21ksoftirqd/100:00:001
17179099537,6cyclictest131rcu_preempt00:39:591
17179099534,13cyclictest23-21ksoftirqd/123:05:001
17179099531,9cyclictest23-21ksoftirqd/121:20:011
17179099531,16cyclictest23-21ksoftirqd/122:00:021
17178799533,19cyclictest175338-21cron20:20:000
171797995222,5cyclictest240812-21latency_hist23:35:013
171795995245,5cyclictest30-21ksoftirqd/200:28:512
171795995215,8cyclictest30-21ksoftirqd/222:20:022
17179099526,10cyclictest23-21ksoftirqd/123:50:001
171790995215,11cyclictest23-21ksoftirqd/122:15:081
171790995214,7cyclictest131rcu_preempt00:15:021
171787995214,16cyclictest16850irq/134-800000021:12:200
171787995210,35cyclictest174315-21cut19:59:590
17179799512,20cyclictest177961-21latency_hist21:10:003
171797995120,9cyclictest37-21ksoftirqd/323:44:553
17179599514,10cyclictest30-21ksoftirqd/221:05:012
17179599513,10cyclictest30-21ksoftirqd/222:30:022
17179599513,10cyclictest30-21ksoftirqd/219:20:012
171795995112,7cyclictest30-21ksoftirqd/223:44:592
171795995110,13cyclictest30-21ksoftirqd/220:40:022
171790995127,11cyclictest23-21ksoftirqd/122:33:121
171790995127,11cyclictest23-21ksoftirqd/122:33:121
171790995116,13cyclictest23-21ksoftirqd/123:35:001
171790995113,10cyclictest23-21ksoftirqd/122:55:221
171797995021,8cyclictest37-21ksoftirqd/322:56:203
17179599506,9cyclictest30-21ksoftirqd/222:15:012
17179599506,9cyclictest30-21ksoftirqd/222:15:002
17179599506,11cyclictest228692-21ssh23:06:522
17179599506,11cyclictest228692-21ssh23:06:512
171795995042,6cyclictest30-21ksoftirqd/223:45:462
171795995042,6cyclictest30-21ksoftirqd/222:39:202
171795995041,6cyclictest291rcuc/200:07:032
171790995030,14cyclictest3467602-21systemd-journal20:15:001
171797994923,8cyclictest37-21ksoftirqd/323:45:043
171797994920,9cyclictest37-21ksoftirqd/322:31:003
171797994920,9cyclictest37-21ksoftirqd/322:31:003
17179599496,9cyclictest30-21ksoftirqd/222:20:032
17179599496,8cyclictest30-21ksoftirqd/200:40:012
17179599493,10cyclictest30-21ksoftirqd/220:14:592
17179599492,13cyclictest30-21ksoftirqd/221:20:002
171795994917,22cyclictest291rcuc/221:50:022
171795994914,5cyclictest131rcu_preempt23:55:012
17179099495,9cyclictest23-21ksoftirqd/122:50:001
171790994933,12cyclictest3467602-21systemd-journal20:35:001
17179099492,5cyclictest131rcu_preempt23:30:021
171790994921,10cyclictest23-21ksoftirqd/121:47:571
171790994916,6cyclictest131rcu_preempt22:35:171
171790994914,6cyclictest131rcu_preempt21:30:021
17179799484,3cyclictest131rcu_preempt21:15:023
171797994827,13cyclictest309-21in:imuxsock00:10:023
17179599487,8cyclictest30-21ksoftirqd/222:08:102
17179599486,8cyclictest30-21ksoftirqd/221:58:092
17179599486,7cyclictest30-21ksoftirqd/219:40:002
17179599482,5cyclictest30-21ksoftirqd/223:28:542
17179599482,5cyclictest30-21ksoftirqd/222:50:432
171795994811,9cyclictest30-21ksoftirqd/223:37:302
171795994811,9cyclictest30-21ksoftirqd/223:37:302
17179099486,9cyclictest23-21ksoftirqd/121:15:021
17179099486,6cyclictest131rcu_preempt00:29:591
17179099484,7cyclictest23-21ksoftirqd/122:10:531
17179099484,7cyclictest23-21ksoftirqd/122:10:521
171790994837,8cyclictest23-21ksoftirqd/123:12:461
171790994835,9cyclictest23-21ksoftirqd/122:05:021
171790994811,9cyclictest23-21ksoftirqd/119:25:011
171787994831,13cyclictest1-21systemd21:10:000
17179599478,10cyclictest30-21ksoftirqd/221:35:292
17179599476,9cyclictest30-21ksoftirqd/220:47:402
17179599476,9cyclictest30-21ksoftirqd/220:47:392
17179599476,7cyclictest30-21ksoftirqd/200:01:162
17179099479,10cyclictest23-21ksoftirqd/122:09:261
171790994735,9cyclictest23-21ksoftirqd/121:35:021
17179099472,4cyclictest131rcu_preempt00:25:001
171790994714,6cyclictest131rcu_preempt20:25:001
171797994632,10cyclictest275-21dbus-daemon19:35:023
171797994615,5cyclictest131rcu_preempt00:39:593
171795994629,13cyclictest3467602-21systemd-journal20:19:392
171795994627,14cyclictest275-21dbus-daemon19:55:022
171795994618,12cyclictest30-21ksoftirqd/220:05:022
17179599461,5cyclictest0-21swapper/200:20:012
17179099466,6cyclictest131rcu_preempt21:00:011
171790994636,8cyclictest23-21ksoftirqd/120:05:001
171790994636,7cyclictest23-21ksoftirqd/121:40:001
171790994636,5cyclictest223506-21latency_hist22:55:031
171790994635,8cyclictest23-21ksoftirqd/120:25:021
171790994631,12cyclictest3467602-21systemd-journal19:17:401
171790994631,12cyclictest212594-21cron22:29:591
17179099461,9cyclictest23-21ksoftirqd/122:40:591
17179099461,9cyclictest23-21ksoftirqd/122:40:581
17179099461,8cyclictest174313-21munin-run20:00:001
171790994616,5cyclictest131rcu_preempt00:35:021
171790994615,6cyclictest23-21ksoftirqd/100:00:201
171790994615,5cyclictest131rcu_preempt19:39:591
17179799453,34cyclictest174344-21unin-run20:00:013
17179599456,7cyclictest30-21ksoftirqd/221:00:012
171795994538,4cyclictest174352-21kworker/2:320:52:422
171795994538,4cyclictest165190-21kworker/2:219:40:512
171795994537,5cyclictest165190-21kworker/2:219:32:172
17179599451,6cyclictest0-21swapper/219:25:022
171790994536,6cyclictest23-21ksoftirqd/121:42:191
171790994532,10cyclictest23-21ksoftirqd/120:10:011
171790994515,7cyclictest23-21ksoftirqd/121:52:081
17179599445,8cyclictest30-21ksoftirqd/220:35:012
171795994411,12cyclictest30-21ksoftirqd/223:34:222
17179099445,9cyclictest23-21ksoftirqd/123:53:171
171790994427,13cyclictest275-21dbus-daemon20:55:011
17178799442,5cyclictest121ksoftirqd/020:44:340
171797994330,11cyclictest37-21ksoftirqd/323:25:033
171797994326,11cyclictest37-21ksoftirqd/321:46:383
17179599433,6cyclictest30-21ksoftirqd/219:30:002
171795994336,4cyclictest158105-21kworker/2:020:23:052
171795994318,17cyclictest291rcuc/223:00:562
17179599431,7cyclictest0-21swapper/219:46:422
17179099439,5cyclictest131rcu_preempt20:20:001
17179099439,13cyclictest23-21ksoftirqd/121:01:141
17179099434,6cyclictest23-21ksoftirqd/123:15:311
171787994328,12cyclictest16950irq/224-800000021:05:010
17178799431,38cyclictest175101-21systemd-run20:15:010
171790994231,8cyclictest23-21ksoftirqd/119:53:381
171797994129,9cyclictest3467602-21systemd-journal00:10:003
171797994125,10cyclictest3467602-21systemd-journal23:20:003
171797994119,16cyclictest3467602-21systemd-journal20:05:003
171797994116,16cyclictest275-21dbus-daemon20:25:003
17179599417,6cyclictest30-21ksoftirqd/220:30:012
17179599415,6cyclictest30-21ksoftirqd/220:10:032
171795994128,10cyclictest174290-21cron19:59:592
17179599411,7cyclictest171807-21latency_hist19:10:012
17179599411,7cyclictest171807-21latency_hist19:10:012
17179099418,6cyclictest23-21ksoftirqd/123:06:541
17179099418,6cyclictest23-21ksoftirqd/123:06:531
171790994131,7cyclictest23-21ksoftirqd/119:29:391
17179099412,5cyclictest131rcu_preempt20:45:011
17178799419,4cyclictest364-21snmpd20:47:050
17178799419,4cyclictest364-21snmpd20:47:040
17178799412,5cyclictest173019-21modprobe19:35:000
171787994123,14cyclictest177150-21sshd20:53:380
171797994028,8cyclictest37-21ksoftirqd/319:45:023
171797994026,10cyclictest37-21ksoftirqd/322:11:413
171797994026,10cyclictest37-21ksoftirqd/322:11:403
171797994025,12cyclictest37-21ksoftirqd/319:30:023
171790994017,18cyclictest131rcu_preempt19:10:021
171790994017,18cyclictest131rcu_preempt19:10:011
171790994011,6cyclictest131rcu_preempt21:10:021
17178799405,7cyclictest12-21ksoftirqd/020:01:390
171787994032,4cyclictest173435-21latency_hist19:40:010
171787994027,10cyclictest175960-21sed20:30:010
171787994026,10cyclictest175633-21munin-run20:25:000
17178799401,35cyclictest173786-21sshd19:49:390
17179799397,9cyclictest37-21ksoftirqd/322:17:153
171797993924,11cyclictest204159-21sed22:10:033
17179099393,8cyclictest23-21ksoftirqd/119:35:021
171787993923,12cyclictest172475-21sshd19:20:380
171787993923,11cyclictest176136-21munin-run20:35:000
171797993827,9cyclictest37-21ksoftirqd/320:50:013
171797993827,9cyclictest37-21ksoftirqd/320:50:003
171797993826,8cyclictest37-21ksoftirqd/300:26:263
171797993811,5cyclictest131rcu_preempt19:55:023
17179099385,7cyclictest23-21ksoftirqd/119:49:401
171790993828,8cyclictest23-21ksoftirqd/119:45:001
171790993812,5cyclictest131rcu_preempt20:40:011
171787993823,11cyclictest3467602-21systemd-journal19:26:400
17179799375,3cyclictest131rcu_preempt20:20:003
17179799372,6cyclictest131rcu_preempt19:20:013
171787993733,2cyclictest0-21swapper/020:08:400
171797993624,9cyclictest176104-21sshd20:31:393
171797993622,11cyclictest37-21ksoftirqd/321:35:033
171797993523,6cyclictest174834-21munin-plugin-st20:10:003
171797993522,7cyclictest309-21in:imuxsock20:40:023
17179099354,6cyclictest0-21swapper/119:12:441
171790993524,9cyclictest23-21ksoftirqd/120:46:401
171790993524,9cyclictest23-21ksoftirqd/120:46:391
17178799354,27cyclictest0-21swapper/019:52:570
171787993522,9cyclictest48-21kcompactd019:40:170
17179799344,23cyclictest3467602-21systemd-journal21:00:003
171797993420,10cyclictest3467602-21systemd-journal20:52:383
17179799335,3cyclictest131rcu_preempt21:05:033
171797993312,15cyclictest171896-21latency_hist19:10:023
171797993312,15cyclictest171896-21latency_hist19:10:013
17179799322,3cyclictest131rcu_preempt20:30:023
171787993120,9cyclictest12-21ksoftirqd/019:10:010
171787993120,9cyclictest12-21ksoftirqd/019:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional