You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 00:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed May 13, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218860991001,3cyclictest0-21swapper/008:40:010
218860999685,3cyclictest0-21swapper/010:30:030
218865999510,58cyclictest316677-21ssh12:20:011
218860998923,14cyclictest230172-21ssh09:20:100
218860998627,54cyclictest320835-21ssh12:28:490
218860998624,14cyclictest256684-21ssh10:15:060
218860998122,17cyclictest282297-21ssh11:08:550
218860997924,11cyclictest278388-21ssh11:00:230
218860997920,14cyclictest269886-21ssh10:42:550
218860997920,14cyclictest269886-21ssh10:42:540
218860997824,34cyclictest259712-21ssh10:21:330
21886099772,69cyclictest316616-21/usr/sbin/munin12:20:010
218860997638,35cyclictest16850irq/134-800000010:05:010
218860997620,11cyclictest239354-21ssh09:39:400
218860997615,11cyclictest272378-21ssh10:48:080
218860997522,9cyclictest275998-21ssh10:55:270
218860997518,4cyclictest16950irq/224-800000012:25:010
218860997428,43cyclictest226972-21ssh09:13:550
21886099742,66cyclictest364-21snmpd12:35:010
21886099742,66cyclictest364-21snmpd12:35:010
218860997327,30cyclictest236796-21ssh09:34:190
21886099732,65cyclictest285244-21latency_hist11:15:020
21886099732,65cyclictest285244-21latency_hist11:15:020
218860997325,14cyclictest309720-21ssh12:05:180
21886099731,55cyclictest325628-21ssh12:38:350
21886099731,54cyclictest291563-21ssh11:27:530
218860997127,26cyclictest313705-21ssh12:13:580
218860997127,26cyclictest313705-21ssh12:13:570
218860997117,49cyclictest16850irq/134-800000009:59:140
218860997065,3cyclictest0-21swapper/007:55:020
218860997025,41cyclictest305898-21ssh11:57:400
21886099701,10cyclictest254051-21wc10:10:020
218860996921,13cyclictest285888-21ssh11:15:580
218860996919,6cyclictest16850irq/134-800000011:20:150
218860996919,6cyclictest16850irq/134-800000011:20:150
21886099682,58cyclictest244369-21sort09:50:020
21886099682,58cyclictest244369-21sort09:50:020
21886099672,5cyclictest275617-21munin-run10:55:020
218860996716,13cyclictest301057-21ssh11:47:360
218860996716,13cyclictest301057-21ssh11:47:360
218860996716,13cyclictest296557-21ssh11:38:260
218860996712,15cyclictest308243-21ssh12:02:220
218860996712,15cyclictest308243-21ssh12:02:210
218860996623,13cyclictest239814-21ssh09:40:040
218860996619,12cyclictest244779-21ssh09:50:280
21886099661,2cyclictest0-21swapper/009:25:040
21886099651,25cyclictest302725-21ssh11:50:570
21887599625,40cyclictest220318-21kworker/u8:007:55:023
21886599622,47cyclictest297396-21ssh11:40:021
218860996227,30cyclictest265595-21ssh10:34:080
218860996220,12cyclictest294353-21ssh11:33:490
218860996220,12cyclictest294353-21ssh11:33:490
218860996130,28cyclictest16950irq/224-800000008:10:020
218860996125,32cyclictest266772-21ssh10:36:180
21886099612,19cyclictest254408-21ssh10:10:220
21887599601,12cyclictest256409-21cut10:15:003
21886099606,19cyclictest222187-21unin-run08:15:010
21886099602,48cyclictest0-21swapper/011:44:300
218860996021,3cyclictest0-21swapper/009:15:110
218860996020,9cyclictest225309-21cat09:10:020
218860996020,9cyclictest225309-21cat09:10:020
218860996016,10cyclictest275-21dbus-daemon07:25:010
218860995920,7cyclictest12-21ksoftirqd/008:20:030
21886099591,26cyclictest0-21swapper/008:50:490
218875995822,10cyclictest37-21ksoftirqd/309:15:013
21887099582,49cyclictest294936-21ssh11:35:022
21887099582,49cyclictest294936-21ssh11:35:022
21886099589,12cyclictest275-21dbus-daemon08:30:010
218870995737,17cyclictest275-21dbus-daemon12:20:012
218875995622,11cyclictest37-21ksoftirqd/312:38:253
218860995626,26cyclictest19350irq/34-mmc108:05:020
218860995617,10cyclictest12-21ksoftirqd/008:45:010
218860995617,10cyclictest12-21ksoftirqd/008:45:000
21887599557,8cyclictest37-21ksoftirqd/310:41:403
21887599557,8cyclictest37-21ksoftirqd/310:41:403
21887099548,6cyclictest131rcu_preempt09:30:012
218870995437,12cyclictest1-21systemd12:05:012
218870995437,12cyclictest1-21systemd12:05:002
218870995425,16cyclictest218855-21cyclictest10:10:032
218865995431,17cyclictest275-21dbus-daemon09:30:011
218870995345,5cyclictest232328-21kworker/2:209:34:272
218870995336,13cyclictest273172-21cron10:50:002
21887099533,46cyclictest251691-21ssh10:05:012
218865995334,14cyclictest3467602-21systemd-journal10:14:591
218865995333,14cyclictest1-21systemd11:10:001
21887599527,6cyclictest131rcu_preempt10:06:493
218875995241,8cyclictest37-21ksoftirqd/309:36:183
218875995221,9cyclictest37-21ksoftirqd/309:30:023
218870995244,5cyclictest223247-21kworker/2:309:17:412
218870995243,6cyclictest275638-21kworker/2:111:08:132
218870995235,13cyclictest3467602-21systemd-journal07:49:432
21886099524,10cyclictest0-21swapper/007:40:000
21887599517,10cyclictest37-21ksoftirqd/310:40:003
218875995141,7cyclictest37-21ksoftirqd/308:41:453
218875995141,7cyclictest37-21ksoftirqd/308:41:443
218875995136,11cyclictest275-21dbus-daemon07:35:003
218870995143,5cyclictest232328-21kworker/2:210:25:142
218870995142,6cyclictest222137-21kworker/2:009:21:472
218870995142,5cyclictest275638-21kworker/2:111:13:352
218870995142,5cyclictest275638-21kworker/2:111:13:352
218865995136,11cyclictest1-21systemd09:00:021
218865995117,6cyclictest131rcu_preempt10:10:021
218860995121,4cyclictest0-21swapper/007:42:460
21887599509,5cyclictest131rcu_preempt11:10:023
218875995037,9cyclictest1-21systemd11:40:013
21887599503,12cyclictest150-21pr/ttyS211:18:433
218875995027,13cyclictest37-21ksoftirqd/311:15:013
218875995027,13cyclictest37-21ksoftirqd/311:15:003
218875995019,12cyclictest0-21swapper/308:15:023
218870995042,5cyclictest307021-21kworker/2:212:38:362
218870995042,5cyclictest275638-21kworker/2:111:04:212
218870995042,5cyclictest241951-21kworker/2:310:30:282
218870995042,5cyclictest241951-21kworker/2:310:20:072
218870995042,5cyclictest232328-21kworker/2:209:47:032
218870995042,5cyclictest232328-21kworker/2:209:47:022
218865995031,15cyclictest1-21systemd08:10:001
218860995027,20cyclictest12-21ksoftirqd/009:05:020
218875994940,7cyclictest37-21ksoftirqd/312:18:093
218870994941,5cyclictest307021-21kworker/2:212:32:472
218870994941,5cyclictest307021-21kworker/2:212:32:462
218870994941,5cyclictest275638-21kworker/2:111:21:172
218870994941,5cyclictest275638-21kworker/2:111:21:172
218870994941,5cyclictest232328-21kworker/2:209:50:372
218870994940,6cyclictest218452-21kworker/2:307:12:062
218865994933,13cyclictest23-21ksoftirqd/108:00:011
218865994928,15cyclictest275-21dbus-daemon11:35:001
218865994928,15cyclictest275-21dbus-daemon11:35:001
218860994924,7cyclictest12-21ksoftirqd/007:50:010
218875994840,5cyclictest37-21ksoftirqd/311:42:123
218875994834,11cyclictest37-21ksoftirqd/312:26:073
218875994817,6cyclictest131rcu_preempt09:20:003
218870994841,5cyclictest275638-21kworker/2:110:57:302
218870994840,5cyclictest307021-21kworker/2:212:22:132
218870994840,5cyclictest275638-21kworker/2:111:15:152
218870994840,5cyclictest232328-21kworker/2:209:41:282
218870994839,5cyclictest222137-21kworker/2:009:12:362
218870994838,6cyclictest275638-21kworker/2:111:40:002
218870994831,13cyclictest268391-21systemctl10:40:002
21886599486,6cyclictest131rcu_preempt08:41:461
21886599486,6cyclictest131rcu_preempt08:41:451
218865994834,10cyclictest23-21ksoftirqd/111:05:021
218865994825,18cyclictest1-21systemd12:15:011
218865994825,18cyclictest1-21systemd12:15:011
218865994814,16cyclictest251747-21ssh10:05:021
21887599474,6cyclictest131rcu_preempt09:48:393
21887599474,6cyclictest131rcu_preempt09:48:383
218875994730,13cyclictest275-21dbus-daemon09:05:013
218875994714,9cyclictest131rcu_preempt09:53:073
218875994710,9cyclictest131rcu_preempt10:27:493
218870994740,5cyclictest307021-21kworker/2:212:29:272
218870994739,5cyclictest307021-21kworker/2:212:05:082
218870994739,5cyclictest280467-21kworker/2:011:28:002
218870994739,5cyclictest275638-21kworker/2:111:41:342
218870994738,5cyclictest232328-21kworker/2:210:40:182
218870994738,5cyclictest232328-21kworker/2:210:40:172
218865994730,13cyclictest1-21systemd10:50:001
218865994730,13cyclictest1-21systemd07:35:001
21886099471,24cyclictest0-21swapper/008:55:410
21887599469,5cyclictest131rcu_preempt10:46:213
21887599466,10cyclictest37-21ksoftirqd/308:20:003
21887599464,8cyclictest131rcu_preempt11:59:393
218875994625,15cyclictest275-21dbus-daemon11:55:003
21887599461,5cyclictest131rcu_preempt12:01:223
21887599461,5cyclictest131rcu_preempt12:01:213
218875994611,12cyclictest37-21ksoftirqd/310:20:463
218870994638,5cyclictest211141-21kworker/2:208:06:502
218870994630,12cyclictest275-21dbus-daemon07:20:012
218865994629,13cyclictest275-21dbus-daemon09:40:001
218860994613,9cyclictest12-21ksoftirqd/007:15:020
21887599457,6cyclictest131rcu_preempt10:30:033
21887599457,5cyclictest131rcu_preempt08:10:013
21887599457,10cyclictest37-21ksoftirqd/312:34:343
21887599457,10cyclictest37-21ksoftirqd/312:34:343
218875994536,6cyclictest37-21ksoftirqd/310:19:423
218875994535,7cyclictest37-21ksoftirqd/311:21:123
218875994535,7cyclictest37-21ksoftirqd/311:21:113
218875994534,8cyclictest37-21ksoftirqd/310:51:413
218875994528,8cyclictest309-21in:imuxsock12:25:013
218875994527,14cyclictest1-21systemd12:10:013
218870994538,5cyclictest232328-21kworker/2:210:50:422
21887099453,3cyclictest131rcu_preempt11:50:322
218865994529,12cyclictest23-21ksoftirqd/107:50:011
218865994527,12cyclictest304537-21systemd-run11:55:001
218860994522,4cyclictest0-21swapper/008:46:160
218860994520,6cyclictest12-21ksoftirqd/008:35:000
21886099451,4cyclictest0-21swapper/007:33:510
218875994435,6cyclictest37-21ksoftirqd/311:02:193
218875994430,8cyclictest223318-21cat08:35:023
218875994421,16cyclictest275-21dbus-daemon07:50:003
218875994420,5cyclictest131rcu_preempt09:22:313
218875994416,9cyclictest37-21ksoftirqd/312:12:173
218875994416,9cyclictest37-21ksoftirqd/312:12:173
21887599441,5cyclictest131rcu_preempt11:31:423
21887599441,5cyclictest131rcu_preempt11:31:423
218875994413,6cyclictest131rcu_preempt11:00:003
218870994436,5cyclictest232328-21kworker/2:209:58:492
218870994436,5cyclictest211141-21kworker/2:208:15:022
218865994429,11cyclictest1-21systemd10:25:001
21886599442,3cyclictest131rcu_preempt09:24:131
218865994420,19cyclictest23-21ksoftirqd/111:49:301
218865994420,19cyclictest23-21ksoftirqd/111:49:291
218865994418,21cyclictest23-21ksoftirqd/112:40:001
218875994326,13cyclictest3467602-21systemd-journal11:30:003
21887599431,8cyclictest0-21swapper/309:43:013
21887599431,7cyclictest0-21swapper/309:34:413
218870994335,5cyclictest275638-21kworker/2:111:48:292
218870994335,5cyclictest275638-21kworker/2:111:48:282
218870994332,8cyclictest30-21ksoftirqd/211:55:332
218870994332,8cyclictest239784-21sed09:40:032
21887099431,5cyclictest0-21swapper/207:40:432
21886599434,7cyclictest23-21ksoftirqd/109:55:001
218865994332,6cyclictest225087-21diskmemload10:20:011
218865994330,7cyclictest23-21ksoftirqd/109:12:071
21886599432,8cyclictest292507-21munin-run11:30:001
218865994326,11cyclictest249172-21cron10:00:001
218865994322,15cyclictest3467602-21systemd-journal09:10:011
218865994322,15cyclictest3467602-21systemd-journal09:10:001
218860994322,4cyclictest0-21swapper/007:19:540
218860994311,18cyclictest311-21rs:main0
21887599427,11cyclictest37-21ksoftirqd/307:30:023
21887599426,6cyclictest131rcu_preempt08:40:003
218875994233,7cyclictest37-21ksoftirqd/307:19:393
218875994228,10cyclictest37-21ksoftirqd/310:02:533
218875994213,5cyclictest131rcu_preempt09:58:463
218875994210,5cyclictest131rcu_preempt09:10:013
218875994210,5cyclictest131rcu_preempt09:10:003
218870994234,5cyclictest211141-21kworker/2:208:03:482
218870994228,10cyclictest314304-21latency_hist12:15:022
218870994228,10cyclictest314304-21latency_hist12:15:022
218865994233,5cyclictest229854-21sh09:20:011
218865994225,12cyclictest23-21ksoftirqd/108:30:001
218865994217,21cyclictest23-21ksoftirqd/112:06:421
218865994214,6cyclictest131rcu_preempt11:15:221
21886099422,7cyclictest221402-21munin-run08:00:000
21887599417,8cyclictest37-21ksoftirqd/308:30:003
21887599415,7cyclictest131rcu_preempt09:00:003
218875994111,8cyclictest131rcu_preempt07:15:013
218870994133,5cyclictest241951-21kworker/2:310:18:172
218870994126,11cyclictest1-21systemd07:30:002
218865994127,9cyclictest0-21swapper/109:45:021
218865994126,11cyclictest1-21systemd08:40:001
218865994122,14cyclictest1-21systemd11:45:011
218865994117,6cyclictest23-21ksoftirqd/110:40:221
218865994117,6cyclictest23-21ksoftirqd/110:40:221
21887599406,8cyclictest37-21ksoftirqd/311:45:253
21887599406,8cyclictest37-21ksoftirqd/311:45:243
218875994024,10cyclictest309-21in:imuxsock07:55:033
21887099407,10cyclictest30-21ksoftirqd/208:35:432
218870994028,10cyclictest30-21ksoftirqd/208:35:012
218870994024,13cyclictest3467602-21systemd-journal07:35:012
218865994028,9cyclictest275-21dbus-daemon11:00:011
218865994026,10cyclictest1-21systemd07:25:011
218865994025,10cyclictest3467602-21systemd-journal08:55:021
218865994023,4cyclictest23-21ksoftirqd/110:35:411
218865994017,5cyclictest23-21ksoftirqd/108:50:001
21886599401,5cyclictest131rcu_preempt12:22:441
218865994012,20cyclictest307187-21latency_hist12:00:021
218860994029,8cyclictest12-21ksoftirqd/008:25:020
21887599396,6cyclictest131rcu_preempt08:03:433
218875993924,11cyclictest37-21ksoftirqd/307:40:013
21887099398,6cyclictest131rcu_preempt08:25:432
218870993931,5cyclictest218452-21kworker/2:307:24:372
218870993928,9cyclictest30-21ksoftirqd/210:14:462
218870993921,12cyclictest1-21systemd07:40:002
21886599398,6cyclictest131rcu_preempt07:45:021
218865993920,12cyclictest275-21dbus-daemon08:15:021
21886599391,7cyclictest263740-21latency_hist10:30:021
21886599391,5cyclictest131rcu_preempt07:30:011
218865993915,16cyclictest23-21ksoftirqd/111:22:521
218865993915,16cyclictest23-21ksoftirqd/111:22:521
218860993922,4cyclictest0-21swapper/008:24:090
21887599388,5cyclictest131rcu_preempt08:55:033
21887599387,5cyclictest131rcu_preempt08:25:013
21887599387,5cyclictest131rcu_preempt08:25:003
21887599384,6cyclictest131rcu_preempt08:50:013
218870993831,5cyclictest211141-21kworker/2:208:19:082
218870993824,11cyclictest3467602-21systemd-journal09:00:012
21887099381,5cyclictest131rcu_preempt09:05:012
218865993829,5cyclictest23-21ksoftirqd/107:55:001
218865993828,8cyclictest23-21ksoftirqd/109:48:571
218865993828,8cyclictest23-21ksoftirqd/109:48:561
218865993826,10cyclictest23-21ksoftirqd/111:11:291
218865993826,10cyclictest23-21ksoftirqd/111:11:281
218865993818,11cyclictest23-21ksoftirqd/107:20:001
218865993814,15cyclictest23-21ksoftirqd/112:27:571
218865993813,16cyclictest23-21ksoftirqd/110:51:181
218870993729,5cyclictest223247-21kworker/2:308:41:522
218870993729,5cyclictest223247-21kworker/2:308:41:522
218870993723,11cyclictest221568-21sed08:00:022
218870993720,12cyclictest275-21dbus-daemon07:55:002
218865993725,8cyclictest3467602-21systemd-journal09:03:431
21886599372,4cyclictest131rcu_preempt09:31:461
218865993724,10cyclictest23-21ksoftirqd/112:31:551
218865993724,10cyclictest23-21ksoftirqd/112:31:541
218865993719,5cyclictest23-21ksoftirqd/110:34:251
21886599371,5cyclictest131rcu_preempt07:15:011
218875993623,9cyclictest309-21in:imuxsock07:21:433
21886599361,3cyclictest131rcu_preempt08:25:031
21886599361,3cyclictest131rcu_preempt08:25:031
218865993610,5cyclictest131rcu_preempt08:02:431
218865993522,10cyclictest23-21ksoftirqd/108:20:011
218865993521,7cyclictest23-21ksoftirqd/112:00:041
218865993521,7cyclictest23-21ksoftirqd/112:00:041
218865993521,10cyclictest223426-21sed08:35:021
218865993520,11cyclictest0-21swapper/107:40:001
21886599351,5cyclictest131rcu_preempt07:10:011
218860993522,8cyclictest218986-21cat07:10:020
21887599345,7cyclictest131rcu_preempt07:10:013
218875993422,9cyclictest37-21ksoftirqd/307:45:023
218870993423,8cyclictest218873-21sort07:10:022
21887099334,23cyclictest225094-21munin-run09:10:012
21887099334,23cyclictest225094-21munin-run09:10:002
218870993318,11cyclictest30-21ksoftirqd/208:24:202
218870993318,11cyclictest30-21ksoftirqd/208:24:192
218870993318,11cyclictest1-21systemd08:55:002
218870993221,9cyclictest30-21ksoftirqd/208:50:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional