You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 04:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 01, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3753153998618,58cyclictest16850irq/134-800000021:30:490
3753153998514,33cyclictest16850irq/134-800000000:30:180
3753153998514,33cyclictest16850irq/134-800000000:30:170
3753153998327,12cyclictest3766972-21ssh21:27:220
3753153998116,19cyclictest16850irq/134-800000023:55:280
3753153997712,33cyclictest16850irq/134-800000023:20:060
3753153997617,54cyclictest16850irq/134-800000021:54:150
3753153997615,35cyclictest16850irq/134-800000023:37:270
3753160997524,14cyclictest3768661-21ssh21:31:082
3753153997519,51cyclictest16850irq/134-800000021:22:430
3753153997516,54cyclictest16850irq/134-800000023:02:030
3753160997417,17cyclictest3779423-21ssh21:56:092
3753153997417,34cyclictest16850irq/134-800000023:10:570
3753153997415,34cyclictest16850irq/134-800000000:04:350
3753153997413,31cyclictest16850irq/134-800000000:27:090
3753160997323,16cyclictest3822307-21ssh23:35:202
3753153997324,33cyclictest12-21ksoftirqd/022:29:090
3753153997324,33cyclictest12-21ksoftirqd/022:29:080
3753153997316,33cyclictest16850irq/134-800000022:52:240
3753153997215,33cyclictest16850irq/134-800000023:40:070
3753153997215,33cyclictest16850irq/134-800000021:37:050
3753153997212,30cyclictest16850irq/134-800000023:28:280
3753153997116,32cyclictest16850irq/134-800000023:54:140
3753153997116,32cyclictest16850irq/134-800000022:38:510
3753153997115,33cyclictest16850irq/134-800000022:45:480
3753153997115,33cyclictest16850irq/134-800000022:45:470
3753153997113,6cyclictest16850irq/134-800000022:05:440
3753153997111,5cyclictest16850irq/134-800000000:11:370
3753163997031,35cyclictest37-21ksoftirqd/320:25:003
3753163997031,35cyclictest37-21ksoftirqd/320:25:003
375316399702,14cyclictest3778500-21ssh21:54:193
3753153997018,33cyclictest16850irq/134-800000022:14:440
3753153997016,36cyclictest16850irq/134-800000000:23:100
3753153997014,33cyclictest16850irq/134-800000021:56:220
3753153997013,34cyclictest16850irq/134-800000022:19:530
3753163996929,36cyclictest3846262-21sh00:31:023
3753163996929,36cyclictest3846262-21sh00:31:023
3753163996927,37cyclictest275-21dbus-daemon23:15:003
3753160996921,15cyclictest3467602-21systemd-journal00:16:132
3753163996829,35cyclictest3783993-21sh22:06:523
3753160996817,16cyclictest3753605-21munin-run19:20:002
3753153996819,43cyclictest16850irq/134-800000022:57:510
3753153996819,43cyclictest16850irq/134-800000022:57:510
3753160996723,14cyclictest3785385-21cat22:10:022
375315399677,31cyclictest16850irq/134-800000023:45:070
375315399677,31cyclictest16850irq/134-800000023:45:060
3753153996715,32cyclictest16850irq/134-800000023:30:480
375316399661,17cyclictest3816031-21ssh23:20:513
3753160996620,14cyclictest1-21systemd20:20:012
375316399652,16cyclictest3821978-21munin-run23:35:013
3753160996522,12cyclictest3849307-21ssh00:38:112
375315399658,31cyclictest16850irq/134-800000021:49:450
3753160996419,14cyclictest3846270-21ssh00:31:032
3753160996419,14cyclictest3846270-21ssh00:31:022
3753153996415,33cyclictest16850irq/134-800000022:33:480
375316399631,15cyclictest3756184-21cron20:10:003
3753160996327,5cyclictest3757254-21munin-run20:30:012
3753160996314,14cyclictest3837509-21ssh00:10:402
375316399622,8cyclictest3798132-21cron22:40:003
375316399622,15cyclictest3765253-21rm21:23:293
375316399622,13cyclictest3756420-21sshd20:11:283
375316399621,16cyclictest3783121-21munin-run22:05:013
375316399621,16cyclictest3780983-21munin-run22:00:013
3753160996243,15cyclictest3825378-21ssh23:42:422
3753160996218,14cyclictest3759320-21diskmemload21:51:472
3753160996218,12cyclictest0-21swapper/222:00:312
375316399618,4cyclictest0-21swapper/321:36:083
375316399612,18cyclictest3757480-21cron20:35:013
375316399612,15cyclictest3822222-21ssh23:35:073
375316399612,13cyclictest48-21kcompactd022:55:053
375316399612,13cyclictest48-21kcompactd022:55:053
375316399612,13cyclictest364-21snmpd19:55:383
375316399611,13cyclictest3759077-21munin-run21:05:013
3753160996124,6cyclictest3835431-21ssh00:06:002
3753160996124,6cyclictest3835431-21ssh00:05:592
3753163996027,29cyclictest3758161-21latency_hist20:45:023
375316399602,14cyclictest364-21snmpd23:57:303
375316399601,7cyclictest3827569-21ssh23:47:533
375316399601,7cyclictest3827569-21ssh23:47:533
3753160996019,13cyclictest3764416-21ssh21:21:142
3753160996019,11cyclictest0-21swapper/223:21:542
3753160996016,13cyclictest3771351-21ssh21:37:312
3753160996011,15cyclictest3831542-21ssh23:57:072
3753163995933,23cyclictest3754378-21munin-plugin-st19:35:003
3753163995920,34cyclictest3791451-21sh22:24:173
3753160995939,16cyclictest3792904-21ssh22:27:292
3753160995939,16cyclictest3792904-21ssh22:27:292
3753160995929,5cyclictest3795775-21ssh22:34:252
3753160995916,12cyclictest0-21swapper/221:26:182
375316399582,11cyclictest48-21kcompactd022:15:343
3753163995819,33cyclictest3794078-21cat22:30:023
3753163995819,33cyclictest3794078-21cat22:30:023
3753163995813,7cyclictest3769750-21ssh21:33:543
3753160995845,10cyclictest3791039-21ssh22:23:192
3753160995845,10cyclictest3774301-21ssh21:44:352
3753160995823,32cyclictest3828381-21modprobe23:50:002
3753160995823,32cyclictest3828381-21modprobe23:50:002
3753160995819,7cyclictest3821641-21ssh23:34:162
3753160995817,13cyclictest364-21snmpd00:01:512
3753160995813,14cyclictest3844336-21ssh00:26:392
3753153995820,34cyclictest16850irq/134-800000000:38:150
3753153995819,34cyclictest16850irq/134-800000021:43:140
3753153995816,21cyclictest16850irq/134-800000021:10:100
3753163995722,31cyclictest37-21ksoftirqd/322:34:483
375316399572,14cyclictest3834892-21unin-run00:05:003
375316399572,11cyclictest364-21snmpd19:47:263
3753163995717,37cyclictest37-21ksoftirqd/323:20:013
375316399571,14cyclictest3848341-21modprobe00:35:443
375316399571,13cyclictest3829134-21ssh23:51:253
375316399571,13cyclictest3810033-21sshd23:07:013
375316399571,13cyclictest3810033-21sshd23:07:013
375315399572,3cyclictest131rcu_preempt21:01:370
375316399563,13cyclictest3844966-21ssh00:28:193
375316399562,17cyclictest3808712-21ssh23:04:163
375316399562,14cyclictest3840104-21ssh00:16:543
375316399562,14cyclictest3803249-21ssh22:51:253
375316399562,14cyclictest3801156-21sh22:46:323
375316399562,14cyclictest3801156-21sh22:46:323
375316399562,14cyclictest3800235-21sh22:44:473
375316399561,13cyclictest364-21snmpd20:01:143
3753163995610,6cyclictest37-21ksoftirqd/321:26:023
3753160995643,10cyclictest3799677-21ssh22:43:172
3753160995639,13cyclictest3808970-21ssh23:04:532
3753160995639,13cyclictest3803955-21ssh22:53:142
3753160995618,6cyclictest3806598-21ssh22:59:242
3753160995618,6cyclictest3806598-21ssh22:59:242
3753156995610,29cyclictest3763663-21modprobe21:20:001
3753153995625,26cyclictest309-21in:imuxsock19:55:010
3753153995619,28cyclictest16850irq/134-800000022:23:280
3753153995618,34cyclictest16850irq/134-800000023:18:270
3753153995618,19cyclictest16850irq/134-800000021:17:580
3753153995617,34cyclictest16850irq/134-800000000:15:390
375316399556,12cyclictest3759055-21kworker/u8:321:41:173
375316399552,6cyclictest364-21snmpd19:17:043
375316399552,21cyclictest3758281-21cut20:50:003
375316399552,18cyclictest3758533-21munin-run20:55:013
375316399552,13cyclictest3838977-21ssh00:14:313
3753163995515,5cyclictest3842882-21sh00:23:253
375316399551,14cyclictest3757726-21sshd20:38:213
375316399551,13cyclictest3825803-21ssh23:43:563
375316399551,13cyclictest3762076-21ssh21:15:573
375316399551,11cyclictest3819543-21ssh23:29:233
3753163995511,15cyclictest275-21dbus-daemon19:45:003
3753160995542,10cyclictest3787288-21ssh22:14:472
3753160995540,11cyclictest3843329-21ssh00:24:402
3753160995539,12cyclictest3819548-21ssh23:29:242
3753160995539,12cyclictest3797748-21ssh22:38:562
3753160995538,14cyclictest3800832-21ssh22:45:452
3753160995538,14cyclictest3800832-21ssh22:45:452
3753160995538,13cyclictest3812725-21ssh23:13:222
3753160995538,13cyclictest3775421-21ssh21:46:552
3753160995537,14cyclictest3761813-21ssh21:15:092
3753160995524,9cyclictest3814090-21ssh23:16:272
3753153995538,13cyclictest1-21systemd19:15:010
3753153995518,33cyclictest16850irq/134-800000022:44:270
3753153995517,19cyclictest16850irq/134-800000023:08:400
3753153995517,19cyclictest16850irq/134-800000023:08:400
3753163995419,31cyclictest3759320-21diskmemload22:10:293
3753160995441,9cyclictest48-21kcompactd019:23:162
3753160995441,10cyclictest3756183-21cron20:10:002
3753160995440,10cyclictest3753303-21sshd19:11:182
3753160995438,13cyclictest3760591-21ssh21:12:392
3753160995438,12cyclictest3830040-21ssh23:53:442
3753160995437,13cyclictest3809481-21ssh23:05:342
3753160995437,13cyclictest3809481-21ssh23:05:332
375316399532,16cyclictest3835760-21ssh00:06:463
375316399532,16cyclictest3835760-21ssh00:06:463
375316399532,14cyclictest3774903-21ssh21:45:403
3753163995319,29cyclictest3693438-1kworker/u9:021:12:003
3753160995336,13cyclictest3787771-21ssh22:15:302
375315699536,43cyclictest3759320-21diskmemload23:00:001
375315699536,43cyclictest3759320-21diskmemload23:00:001
3753153995322,15cyclictest3759346-21ssh21:10:010
3753153995316,33cyclictest16850irq/134-800000022:02:280
3753153995314,35cyclictest16850irq/134-800000000:07:440
3753153995314,35cyclictest16850irq/134-800000000:07:440
375316399523,8cyclictest3759325-21munin-run21:10:003
375316399523,8cyclictest3754619-21cron19:40:003
375316399523,8cyclictest3754619-21cron19:40:003
3753160995239,10cyclictest3755908-21sshd20:00:212
3753163995121,26cyclictest37-21ksoftirqd/319:30:013
375316399511,15cyclictest3756750-21munin-run20:20:013
3753160995137,10cyclictest3758546-21/usr/sbin/munin20:55:022
3753156995118,29cyclictest3761557-21kworker/1:221:30:031
3753160995039,8cyclictest3467602-21systemd-journal20:55:272
3753160995036,10cyclictest364-21snmpd20:49:552
3753163994913,7cyclictest309-21in:imuxsock19:20:013
3753160994933,12cyclictest3755634-21sshd19:56:252
3753160994933,12cyclictest364-21snmpd20:38:092
3753160994933,12cyclictest3467562-21sshd21:06:392
3753156994914,5cyclictest131rcu_preempt23:25:011
3753160994837,8cyclictest3754888-21sshd19:43:202
3753156994834,9cyclictest1-21systemd19:40:011
3753156994834,9cyclictest1-21systemd19:40:011
3753156994831,13cyclictest275-21dbus-daemon23:35:011
375315399481,14cyclictest1-21systemd20:00:010
375316399473,12cyclictest3758808-21grep21:00:003
375316399472,12cyclictest1-21systemd19:15:013
3753160994740,4cyclictest3756427-21sshd20:13:202
3753160994737,7cyclictest3754357-21sshd19:33:202
3753160994735,9cyclictest3756957-21sshd20:21:212
3753160994735,9cyclictest3756957-21sshd20:21:212
375315699473,4cyclictest131rcu_preempt00:25:021
3753156994729,13cyclictest1-21systemd22:45:001
3753153994731,11cyclictest3467602-21systemd-journal20:05:010
375316399463,8cyclictest3757243-21munin-plugin-st20:30:003
3753160994639,4cyclictest3758002-21sshd20:42:212
3753160994638,4cyclictest3754673-21/usr/sbin/munin19:40:012
3753160994638,4cyclictest3754673-21/usr/sbin/munin19:40:012
3753160994638,4cyclictest3754128-21munin-run19:30:002
3753160994632,10cyclictest364-21snmpd21:01:522
3753156994630,12cyclictest3467602-21systemd-journal19:55:001
375315699462,4cyclictest131rcu_preempt23:08:131
375315699462,4cyclictest131rcu_preempt23:08:131
375315699462,4cyclictest131rcu_preempt21:30:011
3753153994630,13cyclictest12-21ksoftirqd/021:00:000
3753153994618,20cyclictest16850irq/134-800000020:06:200
3753160994533,9cyclictest3755132-21sshd19:46:242
3753160994531,11cyclictest364-21snmpd19:51:012
3753156994531,10cyclictest275-21dbus-daemon20:00:011
3753156994529,11cyclictest275-21dbus-daemon22:20:011
3753156994526,14cyclictest23-21ksoftirqd/123:55:011
375315699451,5cyclictest3777756-21ssh21:52:211
375316399443,7cyclictest3755408-21sh19:55:003
3753160994420,19cyclictest3757457-21sshd20:30:212
3753160994416,24cyclictest3753285-21cut19:10:022
375315699447,5cyclictest131rcu_preempt00:33:121
375315699447,5cyclictest131rcu_preempt00:33:121
3753156994429,12cyclictest0-21swapper/122:10:011
3753156994428,13cyclictest23-21ksoftirqd/121:40:001
3753156994427,13cyclictest275-21dbus-daemon00:10:021
3753156994427,13cyclictest275-21dbus-daemon00:10:011
375315399442,6cyclictest3754453-21latency_hist19:35:010
375315399442,4cyclictest131rcu_preempt19:20:000
375315699433,3cyclictest131rcu_preempt23:05:031
3753156994333,7cyclictest3467602-21systemd-journal22:35:021
3753156994329,11cyclictest275-21dbus-daemon21:50:001
3753156994327,13cyclictest1-21systemd22:40:011
3753156994327,11cyclictest1-21systemd20:25:011
3753156994327,11cyclictest1-21systemd20:25:011
375315699432,6cyclictest131rcu_preempt21:25:001
3753153994335,4cyclictest16950irq/224-800000019:40:010
3753153994335,4cyclictest16950irq/224-800000019:40:010
375315399431,38cyclictest3754127-21modprobe19:30:000
375315699424,4cyclictest131rcu_preempt22:29:171
375315699424,4cyclictest131rcu_preempt22:29:171
375315699424,3cyclictest131rcu_preempt22:55:021
375315699424,3cyclictest131rcu_preempt21:55:171
375315699424,28cyclictest3758028-21munin-plugin-st20:45:001
3753156994232,6cyclictest23-21ksoftirqd/100:00:001
3753156994230,9cyclictest23-21ksoftirqd/121:00:001
3753156994230,10cyclictest23-21ksoftirqd/120:15:021
3753156994226,12cyclictest23-21ksoftirqd/119:20:011
375315699421,4cyclictest131rcu_preempt21:44:141
3753153994227,10cyclictest12-21ksoftirqd/020:20:010
375315399422,3cyclictest131rcu_preempt20:30:000
375315399422,35cyclictest1-21systemd20:55:010
375315699415,4cyclictest131rcu_preempt22:03:421
375315699413,6cyclictest131rcu_preempt00:30:011
375315699413,3cyclictest131rcu_preempt00:15:001
3753156994131,5cyclictest3824088-21modprobe23:40:001
3753156994129,9cyclictest23-21ksoftirqd/123:49:211
3753156994129,9cyclictest23-21ksoftirqd/123:49:211
3753156994128,10cyclictest23-21ksoftirqd/122:46:551
3753156994128,10cyclictest23-21ksoftirqd/122:46:551
3753156994127,11cyclictest23-21ksoftirqd/120:55:011
3753156994127,11cyclictest23-21ksoftirqd/100:35:171
3753156994112,23cyclictest3759330-21ssh21:10:001
375315699403,3cyclictest131rcu_preempt00:15:171
3753156994031,5cyclictest3791818-21latency_hist22:25:011
3753156994027,11cyclictest23-21ksoftirqd/120:05:251
375315699402,3cyclictest131rcu_preempt22:10:221
375315699401,3cyclictest131rcu_preempt23:43:121
3753153994023,13cyclictest3757721-21sshd20:37:210
375315699395,3cyclictest131rcu_preempt23:18:201
375315699393,4cyclictest131rcu_preempt23:11:311
3753156993931,5cyclictest3759062-21munin-plugin-st21:05:001
375315699392,4cyclictest131rcu_preempt21:11:241
3753156993923,12cyclictest3467602-21systemd-journal20:40:001
3753156993923,11cyclictest275-21dbus-daemon20:05:001
375315699391,4cyclictest131rcu_preempt20:26:211
3753153993930,6cyclictest12-21ksoftirqd/020:50:020
3753153993926,9cyclictest12-21ksoftirqd/020:15:010
375315699387,3cyclictest131rcu_preempt19:25:561
375315699384,29cyclictest3755009-21cat19:45:011
375315699382,4cyclictest131rcu_preempt00:03:161
375315699381,4cyclictest131rcu_preempt23:28:031
375315399388,3cyclictest131rcu_preempt19:50:010
3753153993824,10cyclictest3754911-21cut19:45:000
3753156993725,9cyclictest23-21ksoftirqd/120:35:011
3753153993727,6cyclictest151rcuc/020:35:010
375315399372,26cyclictest3758030-21munin-plugin-st20:45:000
3753156993627,6cyclictest309-21in:imuxsock20:20:001
3753156993626,7cyclictest23-21ksoftirqd/119:10:021
3753156993624,9cyclictest23-21ksoftirqd/119:20:551
375315399363,26cyclictest3753874-21modprobe19:25:000
3753153993619,12cyclictest275-21dbus-daemon20:25:010
3753153993619,12cyclictest275-21dbus-daemon20:25:010
375315699357,23cyclictest3753329-21sh19:15:001
375315699354,21cyclictest3754399-21latency_hist19:35:011
3753156993523,9cyclictest275-21dbus-daemon19:50:011
3753156993427,4cyclictest3758275-21latency_hist20:50:021
3753163992721,4cyclictest3753077-21latency_hist19:10:023
3753153992620,4cyclictest3753077-21latency_hist19:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional