You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 14:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Apr 17, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23208099124105,10cyclictest16950irq/224-800000022:30:010
232080997910,46cyclictest12-21ksoftirqd/023:05:040
23209299761,52cyclictest0-21swapper/321:50:023
232092997429,17cyclictest233818-21sshd19:41:543
23208599725,48cyclictest311734-21ssh00:05:031
232092996826,39cyclictest251721-1kworker/u9:221:50:013
232092996717,14cyclictest312749-21ssh00:07:453
232092996517,13cyclictest322980-21ssh00:32:003
232080996516,8cyclictest291692-21ssh23:17:330
232080996421,39cyclictest253003-21sed21:45:030
232092996315,16cyclictest286518-21cut23:05:023
232092996314,14cyclictest273685-21ssh22:34:443
232092996314,14cyclictest273685-21ssh22:34:443
232080996336,10cyclictest308144-21ssh23:56:380
232080996336,10cyclictest308144-21ssh23:56:370
232080996119,7cyclictest12-21ksoftirqd/022:38:090
23209299609,20cyclictest299322-21ssh23:35:273
232092996020,11cyclictest0-21swapper/322:00:563
23208599592,8cyclictest23-21ksoftirqd/100:20:001
232080995922,6cyclictest16850irq/134-800000000:10:020
232080995917,9cyclictest12-21ksoftirqd/021:35:580
232092995843,12cyclictest279595-21ssh22:48:423
232092995839,15cyclictest234588-21sshd19:59:223
23208599586,10cyclictest23-21ksoftirqd/100:30:051
23208099581,37cyclictest238257-21diskmemload00:15:110
232080995812,9cyclictest12-21ksoftirqd/023:26:000
232092995716,8cyclictest287175-21ssh23:06:293
232088995745,7cyclictest236731-21kworker/2:222:31:192
232088995745,7cyclictest236731-21kworker/2:222:31:182
232080995724,30cyclictest12-21ksoftirqd/021:23:490
232080995720,32cyclictest16850irq/134-800000000:38:260
232080995713,37cyclictest16850irq/134-800000023:35:000
232080995713,37cyclictest16850irq/134-800000023:35:000
232092995638,15cyclictest322066-21cat00:30:013
232092995619,6cyclictest308168-21ssh23:56:413
232092995619,6cyclictest308168-21ssh23:56:403
232092995618,11cyclictest0-21swapper/322:39:403
232092995612,10cyclictest317961-21ssh00:20:013
232088995648,5cyclictest236731-21kworker/2:223:09:282
232088995646,6cyclictest218395-21kworker/2:121:10:022
23208099568,9cyclictest12-21ksoftirqd/023:10:060
232080995612,39cyclictest12-21ksoftirqd/023:55:010
232092995541,10cyclictest303090-21ssh23:44:533
232092995539,13cyclictest265640-21ssh22:15:053
232092995536,15cyclictest310018-21ssh00:00:543
232092995517,15cyclictest238257-21diskmemload22:42:503
232092995516,12cyclictest0-21swapper/321:10:263
232092995512,13cyclictest262729-21ssh22:08:303
232088995547,5cyclictest236731-21kworker/2:223:00:022
232088995546,5cyclictest236731-21kworker/2:223:34:152
232088995546,5cyclictest236731-21kworker/2:223:34:152
232085995540,11cyclictest3467602-21systemd-journal23:55:011
232085995518,8cyclictest23-21ksoftirqd/122:51:381
232085995516,20cyclictest286540-21ssh23:05:021
232085995513,9cyclictest23-21ksoftirqd/100:00:021
232085995513,9cyclictest23-21ksoftirqd/100:00:011
23208099559,8cyclictest12-21ksoftirqd/021:53:520
23208099559,7cyclictest12-21ksoftirqd/000:04:260
232080995515,22cyclictest12-21ksoftirqd/022:44:330
232080995511,9cyclictest12-21ksoftirqd/022:34:520
232080995511,9cyclictest12-21ksoftirqd/022:34:510
232092995439,12cyclictest295916-21ssh23:27:303
232092995438,12cyclictest246652-21munin-run21:30:013
232092995436,15cyclictest364-21snmpd20:05:173
232092995436,15cyclictest364-21snmpd20:05:173
232092995434,16cyclictest268604-21ssh22:22:223
232092995434,16cyclictest268604-21ssh22:22:223
232088995445,5cyclictest218395-21kworker/2:121:43:052
232088995444,7cyclictest218395-21kworker/2:121:14:542
232088995444,5cyclictest236731-21kworker/2:222:53:132
232085995413,8cyclictest23-21ksoftirqd/100:35:461
232085995410,7cyclictest23-21ksoftirqd/123:39:081
232080995419,6cyclictest16850irq/134-800000022:14:490
232080995419,6cyclictest16850irq/134-800000022:14:490
232080995415,8cyclictest12-21ksoftirqd/022:51:130
232080995414,9cyclictest12-21ksoftirqd/000:29:040
23208099541,30cyclictest0-21swapper/000:10:410
232080995411,9cyclictest12-21ksoftirqd/021:25:530
232092995340,10cyclictest3467602-21systemd-journal22:51:513
232092995338,10cyclictest233066-21munin-run19:30:023
232092995337,12cyclictest270013-21ssh22:25:373
232092995337,12cyclictest249264-21ssh21:35:543
232092995337,12cyclictest240979-21ssh21:16:073
232092995337,12cyclictest240979-21ssh21:16:063
232092995336,13cyclictest282564-21ssh22:55:283
232092995336,13cyclictest264291-21ssh22:12:073
232092995336,13cyclictest264291-21ssh22:12:063
232092995334,16cyclictest243302-21ssh21:21:443
232092995333,16cyclictest289330-21ssh23:11:423
232092995333,16cyclictest251102-21ssh21:40:083
232092995332,18cyclictest364-21snmpd23:52:543
232092995332,16cyclictest291234-21ssh23:16:153
232088995345,5cyclictest236731-21kworker/2:223:25:032
232088995345,5cyclictest218395-21kworker/2:122:38:402
232088995345,5cyclictest218395-21kworker/2:119:10:022
232088995344,6cyclictest218395-21kworker/2:120:55:022
232088995344,5cyclictest236731-21kworker/2:223:59:592
232088995344,5cyclictest236731-21kworker/2:200:00:002
232088995343,6cyclictest236731-21kworker/2:200:21:412
232088995343,6cyclictest236731-21kworker/2:200:21:412
232085995324,17cyclictest23-21ksoftirqd/122:59:191
23208099537,13cyclictest303780-21ssh23:46:030
232080995335,12cyclictest232078-21cat19:10:010
232080995316,7cyclictest16850irq/134-800000022:00:130
232080995315,33cyclictest16850irq/134-800000023:37:500
232092995238,11cyclictest258051-21ssh21:57:153
232092995238,11cyclictest235692-21munin-run20:20:013
232092995236,13cyclictest318832-21ssh00:22:053
232092995236,13cyclictest318832-21ssh00:22:053
232092995236,13cyclictest294330-21ssh23:23:593
232092995236,12cyclictest247162-21ssh21:30:453
232088995244,5cyclictest236731-21kworker/2:200:35:352
232088995243,6cyclictest294557-21kworker/2:023:50:412
232088995243,6cyclictest236731-21kworker/2:223:44:392
232088995243,6cyclictest236731-21kworker/2:222:01:292
232088995234,13cyclictest259078-21cron21:59:592
23208099528,6cyclictest284912-21ssh23:01:090
232080995218,6cyclictest16850irq/134-800000023:41:560
232080995218,19cyclictest16850irq/134-800000021:48:230
232080995217,6cyclictest16850irq/134-800000022:20:160
232080995217,6cyclictest16850irq/134-800000022:20:150
232080995217,6cyclictest16850irq/134-800000022:06:510
232080995217,5cyclictest16850irq/134-800000021:10:490
232080995217,31cyclictest16850irq/134-800000021:56:310
232080995216,6cyclictest16850irq/134-800000022:45:540
232080995211,8cyclictest12-21ksoftirqd/021:31:020
232092995139,9cyclictest236406-21sshd20:32:223
232092995138,9cyclictest314521-21ssh00:11:533
232092995138,10cyclictest238238-21sshd21:09:233
232092995137,12cyclictest303595-21ssh23:45:363
232092995132,16cyclictest297212-21ssh23:30:313
232092995132,16cyclictest297212-21ssh23:30:313
232088995144,5cyclictest218395-21kworker/2:121:24:272
232088995143,5cyclictest236731-21kworker/2:223:15:462
232088995142,6cyclictest236731-21kworker/2:200:08:432
232088995141,6cyclictest236731-21kworker/2:222:30:012
23208599514,9cyclictest23-21ksoftirqd/120:01:561
23208599512,10cyclictest23-21ksoftirqd/122:39:071
232080995116,6cyclictest16850irq/134-800000000:23:080
232080995116,6cyclictest16850irq/134-800000000:23:070
232080995116,32cyclictest16850irq/134-800000022:56:050
232092995038,9cyclictest235896-21sshd20:21:223
232092995037,10cyclictest324478-21ssh00:35:273
232092995036,11cyclictest236150-21sshd20:27:003
232092995035,12cyclictest232512-21sshd19:18:223
232092995034,13cyclictest237690-21sshd20:57:063
232088995041,6cyclictest218395-21kworker/2:121:46:212
232088995041,5cyclictest236731-21kworker/2:223:49:022
23208599501,9cyclictest23-21ksoftirqd/100:22:281
23208599501,9cyclictest23-21ksoftirqd/100:22:271
232085995010,8cyclictest23-21ksoftirqd/122:32:241
232085995010,8cyclictest23-21ksoftirqd/122:32:231
232080995015,6cyclictest16850irq/134-800000022:18:390
232080995015,6cyclictest16850irq/134-800000021:18:320
232080995015,6cyclictest16850irq/134-800000021:18:310
232092994936,10cyclictest234327-21sshd19:52:233
232092994936,10cyclictest234327-21sshd19:52:223
232088994942,4cyclictest236731-21kworker/2:223:14:492
232088994941,6cyclictest236731-21kworker/2:223:30:032
232088994941,5cyclictest236731-21kworker/2:200:26:532
232088994941,5cyclictest236731-21kworker/2:200:13:282
232088994941,5cyclictest218395-21kworker/2:122:46:142
232088994941,5cyclictest218395-21kworker/2:121:35:182
232088994941,5cyclictest218395-21kworker/2:120:55:012
232088994941,5cyclictest218395-21kworker/2:120:41:222
232088994939,6cyclictest236731-21kworker/2:222:09:062
232088994930,13cyclictest3467602-21systemd-journal22:25:012
232088994930,13cyclictest3467602-21systemd-journal22:25:002
23208599499,8cyclictest23-21ksoftirqd/122:49:021
23208599497,12cyclictest23-21ksoftirqd/121:10:271
23208599496,8cyclictest23-21ksoftirqd/121:56:561
23208599496,10cyclictest23-21ksoftirqd/120:47:221
232085994941,5cyclictest221rcuc/122:17:131
232085994940,6cyclictest221rcuc/122:26:281
23208599492,8cyclictest23-21ksoftirqd/100:07:391
232085994919,9cyclictest131rcu_preempt23:07:081
232080994915,4cyclictest131rcu_preempt20:30:020
232092994836,9cyclictest48-21kcompactd019:36:023
232092994836,9cyclictest236724-21munin-run20:40:013
232092994836,9cyclictest232796-21sshd19:24:223
232092994835,9cyclictest233297-21sshd19:33:223
232092994834,11cyclictest232242-21sshd19:11:233
232092994832,11cyclictest48-21kcompactd020:13:123
232088994840,5cyclictest236731-21kworker/2:222:12:462
232088994840,5cyclictest236731-21kworker/2:222:12:452
232088994840,5cyclictest236731-21kworker/2:200:31:552
232088994839,6cyclictest294557-21kworker/2:023:38:562
232088994839,6cyclictest236731-21kworker/2:222:16:462
232088994839,6cyclictest218395-21kworker/2:121:27:282
232088994839,5cyclictest236731-21kworker/2:221:52:452
232088994838,6cyclictest218395-21kworker/2:121:33:212
232088994838,6cyclictest218395-21kworker/2:119:40:002
232088994833,9cyclictest236693-21grep20:40:002
23208599489,28cyclictest238366-21ssh21:10:031
23208599485,8cyclictest23-21ksoftirqd/121:40:361
232085994821,6cyclictest221rcuc/100:27:221
232085994810,7cyclictest23-21ksoftirqd/122:00:231
232080994818,6cyclictest16850irq/134-800000023:23:070
232088994739,5cyclictest218395-21kworker/2:122:43:422
232088994727,13cyclictest309-21in:imuxsock19:30:002
23208599476,9cyclictest23-21ksoftirqd/122:05:591
23208599474,7cyclictest23-21ksoftirqd/122:14:471
23208599474,7cyclictest23-21ksoftirqd/122:14:461
23208599474,6cyclictest23-21ksoftirqd/123:30:241
23208599474,6cyclictest23-21ksoftirqd/123:30:241
232085994737,6cyclictest278012-21latency_hist22:45:011
23208099474,39cyclictest235685-21true20:20:000
232080994740,4cyclictest16950irq/224-800000019:40:010
232092994632,11cyclictest236928-21sshd20:40:213
232092994631,10cyclictest364-21snmpd20:50:553
232088994641,3cyclictest236731-21kworker/2:200:03:232
232088994638,5cyclictest236731-21kworker/2:223:01:192
232088994638,5cyclictest218395-21kworker/2:121:15:332
232088994638,5cyclictest218395-21kworker/2:121:15:322
23208599467,7cyclictest23-21ksoftirqd/123:25:071
23208599465,7cyclictest23-21ksoftirqd/121:21:151
23208599463,7cyclictest23-21ksoftirqd/121:47:411
232085994610,7cyclictest221rcuc/121:52:041
23208099468,4cyclictest131rcu_preempt20:55:010
232080994612,9cyclictest19350irq/34-mmc119:55:030
232080994612,9cyclictest19350irq/34-mmc119:55:020
232080994612,17cyclictest16850irq/134-800000000:32:440
232092994533,9cyclictest3467602-21systemd-journal20:00:083
232092994532,10cyclictest48-21kcompactd021:03:403
232092994531,11cyclictest234070-21sshd19:46:543
232092994531,10cyclictest0-21swapper/320:46:393
232088994537,5cyclictest218395-21kworker/2:120:09:012
232088994537,5cyclictest218395-21kworker/2:120:09:002
232088994536,6cyclictest218395-21kworker/2:120:25:032
23208599458,5cyclictest131rcu_preempt21:17:221
23208599458,5cyclictest131rcu_preempt21:17:211
23208599456,11cyclictest23-21ksoftirqd/119:30:011
23208599454,8cyclictest23-21ksoftirqd/123:46:041
23208599454,5cyclictest23-21ksoftirqd/121:28:481
23208599453,8cyclictest23-21ksoftirqd/123:12:321
23208599453,8cyclictest221rcuc/100:10:181
23208599453,10cyclictest23-21ksoftirqd/119:25:011
232085994511,4cyclictest131rcu_preempt19:20:021
23208099455,7cyclictest12-21ksoftirqd/020:39:220
232080994538,4cyclictest16950irq/224-800000021:10:010
23208099452,11cyclictest237720-21sh20:59:590
232080994510,6cyclictest131rcu_preempt20:50:000
232088994436,5cyclictest236731-21kworker/2:200:16:172
232088994436,5cyclictest218395-21kworker/2:121:00:262
232088994436,5cyclictest218395-21kworker/2:120:47:232
232088994436,5cyclictest218395-21kworker/2:119:24:392
232088994435,6cyclictest218395-21kworker/2:120:01:252
23208599446,6cyclictest23-21ksoftirqd/122:23:471
23208599446,6cyclictest23-21ksoftirqd/122:23:461
23208599445,6cyclictest23-21ksoftirqd/121:32:391
23208599444,9cyclictest23-21ksoftirqd/121:05:001
23208599444,4cyclictest131rcu_preempt23:41:431
23208599444,12cyclictest23-21ksoftirqd/121:39:151
23208599441,11cyclictest292056-21ssh23:18:381
232080994434,6cyclictest16950irq/224-800000019:25:020
232080994427,13cyclictest3467602-21systemd-journal20:03:230
23208599435,8cyclictest23-21ksoftirqd/120:25:001
23208599433,8cyclictest23-21ksoftirqd/123:22:091
23208099439,30cyclictest236427-21latency_hist20:35:020
232080994336,4cyclictest16950irq/224-800000019:45:010
232080994325,14cyclictest3467718-21lldpd19:28:340
232088994234,5cyclictest218395-21kworker/2:120:16:542
232088994234,5cyclictest218395-21kworker/2:119:40:162
232088994233,5cyclictest218395-21kworker/2:119:55:232
232088994226,10cyclictest275-21dbus-daemon19:55:012
232088994226,10cyclictest275-21dbus-daemon19:55:012
23208599426,6cyclictest23-21ksoftirqd/121:00:011
232085994224,14cyclictest275-21dbus-daemon20:20:001
232080994228,11cyclictest3467602-21systemd-journal19:20:000
232088994133,5cyclictest218395-21kworker/2:120:11:582
23208599418,7cyclictest0-21swapper/120:51:211
23208599414,9cyclictest23-21ksoftirqd/120:10:021
23208599414,9cyclictest23-21ksoftirqd/120:10:011
23208599414,7cyclictest23-21ksoftirqd/119:35:001
23208099419,5cyclictest131rcu_preempt19:55:150
23208099416,7cyclictest234072-21sshd19:46:540
232080994124,13cyclictest12-21ksoftirqd/019:35:010
23208099411,11cyclictest0-21swapper/020:43:220
232088994022,12cyclictest275-21dbus-daemon20:35:022
23208599405,8cyclictest23-21ksoftirqd/120:35:011
232085994025,12cyclictest234380-21cat19:55:021
232085994025,12cyclictest234380-21cat19:55:011
232080994028,8cyclictest232245-21sshd19:11:520
232088993923,12cyclictest3467602-21systemd-journal19:20:002
23208599395,9cyclictest23-21ksoftirqd/119:45:021
23208599392,9cyclictest23-21ksoftirqd/120:35:221
232085993929,7cyclictest23-21ksoftirqd/120:00:001
232085993924,11cyclictest275-21dbus-daemon19:50:001
23208099398,9cyclictest12-21ksoftirqd/020:24:230
232080993925,10cyclictest235119-21sshd20:09:230
232080993925,10cyclictest235119-21sshd20:09:220
232092993824,11cyclictest232108-21latency_hist19:10:013
23208599384,7cyclictest23-21ksoftirqd/120:41:231
23208599384,7cyclictest23-21ksoftirqd/120:14:231
23208599371,9cyclictest0-21swapper/120:28:031
232080993723,10cyclictest235351-21sshd20:10:220
232088993628,5cyclictest218395-21kworker/2:120:30:002
232085993623,10cyclictest1-21systemd19:15:011
23208099362,6cyclictest3467602-21systemd-journal21:03:420
232088993527,5cyclictest218395-21kworker/2:119:10:072
23208599351,6cyclictest23-21ksoftirqd/119:35:121
232088993427,5cyclictest218395-21kworker/2:119:48:232
232088993318,11cyclictest233524-21sed19:35:022
232085992616,7cyclictest232209-21cut19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional