You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 16:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 24, 2026 12:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146350599763,28cyclictest1469153-21kworker/u8:209:55:022
1463496997557,12cyclictest0-21swapper/011:30:030
1463496997557,12cyclictest0-21swapper/011:30:020
146349699746,33cyclictest0-21swapper/012:20:030
146349699724,36cyclictest0-21swapper/010:59:160
1463496997159,8cyclictest16850irq/134-800000011:45:010
1463496997012,24cyclictest1510813-21ssh10:45:170
1463496997012,24cyclictest1510813-21ssh10:45:170
1463496996934,4cyclictest1469734-21munin-run09:10:010
1463496996934,4cyclictest1469734-21munin-run09:10:010
146349699685,41cyclictest1522350-21ssh11:12:360
1463496996814,19cyclictest1474697-21ssh09:21:120
146349699674,32cyclictest1465613-21latency_hist07:50:020
1463496996713,15cyclictest1504893-21ssh10:31:380
1463496996710,25cyclictest1547490-21ssh12:11:200
1463496996614,20cyclictest1492109-21ssh10:01:420
146351099654,14cyclictest37-21ksoftirqd/307:40:003
146351099654,14cyclictest37-21ksoftirqd/307:40:003
1463496996522,15cyclictest1479792-21sh09:33:220
1463510996421,9cyclictest37-21ksoftirqd/312:35:013
1463505996417,13cyclictest30-21ksoftirqd/211:28:052
1463505996417,13cyclictest30-21ksoftirqd/211:28:042
1463501996419,6cyclictest131rcu_preempt10:40:021
1463501996419,6cyclictest131rcu_preempt10:40:011
1463496996429,31cyclictest1491079-21ssh09:59:460
1463496996429,31cyclictest1491079-21ssh09:59:460
1463496996421,39cyclictest3467602-21systemd-journal12:34:450
1463496996416,18cyclictest1553794-21ssh12:26:050
1463496996414,14cyclictest1495682-21cat10:10:020
146351099638,11cyclictest37-21ksoftirqd/309:20:043
146349699635,31cyclictest0-21swapper/011:40:010
1463496996332,28cyclictest1558677-21ssh12:37:440
146351099627,6cyclictest37-21ksoftirqd/309:40:013
146351099627,6cyclictest37-21ksoftirqd/309:40:003
1463510996230,8cyclictest37-21ksoftirqd/308:09:123
146351099622,12cyclictest37-21ksoftirqd/309:35:023
146350199628,5cyclictest131rcu_preempt09:35:021
1463496996247,11cyclictest3467602-21systemd-journal08:40:000
146349699624,23cyclictest3467602-21systemd-journal09:30:000
1463496996231,28cyclictest1527402-21ssh11:24:400
1463496996222,23cyclictest1531810-21ssh11:34:590
1463496996219,7cyclictest1513357-21ssh10:51:210
1463496996216,15cyclictest1488164-21ssh09:52:430
1463496996215,15cyclictest1466221-21date08:00:020
146351099617,9cyclictest37-21ksoftirqd/307:25:023
146351099617,9cyclictest37-21ksoftirqd/307:25:023
146351099615,11cyclictest37-21ksoftirqd/312:10:023
146351099615,11cyclictest37-21ksoftirqd/312:10:023
146351099614,10cyclictest37-21ksoftirqd/309:27:193
146351099612,9cyclictest37-21ksoftirqd/310:40:023
146351099612,9cyclictest37-21ksoftirqd/310:40:013
146351099612,17cyclictest37-21ksoftirqd/309:50:043
146350599612,9cyclictest30-21ksoftirqd/207:40:012
146350599612,9cyclictest30-21ksoftirqd/207:40:012
1463501996113,6cyclictest131rcu_preempt08:05:011
1463501996113,6cyclictest131rcu_preempt08:05:001
1463496996124,4cyclictest3467602-21systemd-journal10:20:000
1463496996116,10cyclictest1496447-21ssh10:12:020
1463496996111,19cyclictest1464971-21latency_hist07:35:030
1463510996017,13cyclictest37-21ksoftirqd/310:15:013
1463510996010,5cyclictest131rcu_preempt12:25:003
146350199605,9cyclictest23-21ksoftirqd/107:50:031
1463501996018,6cyclictest131rcu_preempt11:05:011
1463501996013,6cyclictest131rcu_preempt12:05:021
1463496996029,27cyclictest1473202-21ssh09:17:540
1463496996029,27cyclictest1473202-21ssh09:17:530
1463510995911,6cyclictest131rcu_preempt09:50:003
1463510995911,6cyclictest131rcu_preempt09:50:003
1463505995940,14cyclictest1-21systemd09:10:022
1463505995940,14cyclictest1-21systemd09:10:012
1463501995913,5cyclictest131rcu_preempt11:10:031
1463496995950,6cyclictest16950irq/224-800000010:30:010
1463496995950,6cyclictest16950irq/224-800000010:30:000
1463496995912,13cyclictest1466842-21latency_hist08:15:010
1463510995813,5cyclictest131rcu_preempt09:57:513
1463510995813,5cyclictest131rcu_preempt09:57:513
1463510995811,5cyclictest131rcu_preempt11:20:403
1463510995811,12cyclictest37-21ksoftirqd/311:34:593
146350599588,6cyclictest131rcu_preempt09:35:022
1463505995842,7cyclictest1493353-21kworker/2:110:42:282
1463501995812,6cyclictest131rcu_preempt10:50:011
1463501995812,6cyclictest131rcu_preempt10:50:011
1463496995846,8cyclictest16850irq/134-800000010:40:000
1463496995846,8cyclictest16850irq/134-800000010:40:000
1463496995829,26cyclictest1539977-21ssh11:54:040
1463496995811,15cyclictest1482975-21ssh09:40:170
146351099579,4cyclictest131rcu_preempt10:18:383
146351099577,8cyclictest37-21ksoftirqd/311:51:193
146351099573,7cyclictest37-21ksoftirqd/310:33:193
146351099573,12cyclictest37-21ksoftirqd/309:40:513
1463510995728,9cyclictest37-21ksoftirqd/311:01:093
146351099571,9cyclictest37-21ksoftirqd/310:07:293
1463510995717,5cyclictest131rcu_preempt10:00:493
1463510995715,6cyclictest131rcu_preempt11:13:523
1463510995710,5cyclictest131rcu_preempt07:35:013
1463505995749,5cyclictest1493353-21kworker/2:110:34:552
1463505995742,6cyclictest1467398-21kworker/2:310:19:492
146350199579,7cyclictest131rcu_preempt09:24:321
146350199577,6cyclictest131rcu_preempt11:16:171
1463501995733,18cyclictest0-21swapper/110:20:011
1463496995725,16cyclictest1545822-21ssh12:07:370
1463496995725,16cyclictest1545822-21ssh12:07:360
1463496995717,36cyclictest1502042-21date10:25:010
146351099567,10cyclictest37-21ksoftirqd/311:48:183
146351099566,10cyclictest37-21ksoftirqd/311:00:033
1463510995616,6cyclictest131rcu_preempt11:16:183
1463505995610,6cyclictest131rcu_preempt10:02:152
146350199566,6cyclictest131rcu_preempt11:30:001
146350199566,6cyclictest131rcu_preempt11:30:001
146350199563,5cyclictest131rcu_preempt09:39:451
146350199563,5cyclictest131rcu_preempt09:39:441
146349699564,43cyclictest1542480-21unin-run12:00:000
1463496995626,15cyclictest1508608-21ssh10:40:050
1463496995614,37cyclictest1519132-21latency_hist11:05:010
1463496995612,11cyclictest1542857-21ssh12:00:260
146351099556,13cyclictest37-21ksoftirqd/308:15:013
146351099554,9cyclictest37-21ksoftirqd/311:37:073
146351099551,9cyclictest1474033-21modprobe09:20:003
146351099551,9cyclictest1474033-21modprobe09:20:003
1463510995516,5cyclictest131rcu_preempt11:07:463
1463510995510,6cyclictest131rcu_preempt10:45:323
1463510995510,6cyclictest131rcu_preempt10:45:313
1463505995541,6cyclictest1463100-21kworker/2:008:30:092
1463505995541,6cyclictest1463100-21kworker/2:008:30:092
146350599551,5cyclictest131rcu_preempt09:30:012
146350199557,11cyclictest23-21ksoftirqd/109:12:441
1463501995514,6cyclictest131rcu_preempt11:52:001
1463496995521,30cyclictest1486879-21latency_hist09:50:020
1463496995521,30cyclictest1486879-21latency_hist09:50:020
146351099548,9cyclictest37-21ksoftirqd/311:41:053
146351099545,11cyclictest37-21ksoftirqd/307:50:023
146351099542,9cyclictest37-21ksoftirqd/312:27:083
1463510995412,5cyclictest131rcu_preempt12:15:093
1463510995410,5cyclictest131rcu_preempt12:02:263
1463510995410,5cyclictest131rcu_preempt11:58:043
1463510995410,5cyclictest131rcu_preempt09:10:003
1463510995410,5cyclictest131rcu_preempt09:10:003
146350599543,10cyclictest30-21ksoftirqd/208:00:002
146350199549,6cyclictest131rcu_preempt09:58:191
146350199549,6cyclictest131rcu_preempt09:58:191
146350199548,5cyclictest131rcu_preempt12:37:211
146350199546,9cyclictest23-21ksoftirqd/111:24:271
146350199545,9cyclictest23-21ksoftirqd/110:53:051
1463501995433,15cyclictest3467602-21systemd-journal07:55:001
1463501995413,5cyclictest131rcu_preempt10:56:391
1463501995411,6cyclictest131rcu_preempt12:26:351
1463501995411,5cyclictest131rcu_preempt07:20:001
1463501995410,6cyclictest131rcu_preempt12:18:531
146351099538,10cyclictest37-21ksoftirqd/308:30:003
146351099537,7cyclictest37-21ksoftirqd/311:25:153
146351099537,7cyclictest37-21ksoftirqd/311:25:143
146351099536,8cyclictest37-21ksoftirqd/312:35:093
1463510995312,6cyclictest131rcu_preempt10:43:113
1463510995312,6cyclictest131rcu_preempt10:26:273
1463510995312,6cyclictest131rcu_preempt10:26:263
146350599536,8cyclictest30-21ksoftirqd/207:45:002
1463505995345,5cyclictest1467398-21kworker/2:311:36:072
1463505995344,6cyclictest1467398-21kworker/2:311:34:272
1463505995339,7cyclictest1493353-21kworker/2:111:06:552
1463505995339,6cyclictest1493353-21kworker/2:111:15:442
1463505995339,6cyclictest1493353-21kworker/2:110:38:562
1463505995339,6cyclictest1493353-21kworker/2:110:38:562
1463505995338,7cyclictest1467398-21kworker/2:312:14:052
1463505995338,10cyclictest30-21ksoftirqd/211:21:032
1463505995336,8cyclictest30-21ksoftirqd/209:50:012
1463505995336,8cyclictest30-21ksoftirqd/209:50:002
1463505995335,14cyclictest1-21systemd09:00:012
1463505995328,10cyclictest30-21ksoftirqd/209:40:002
1463505995328,10cyclictest30-21ksoftirqd/209:40:002
146350199539,6cyclictest131rcu_preempt09:41:311
146350199538,6cyclictest131rcu_preempt11:57:231
146350199538,5cyclictest131rcu_preempt11:40:071
146350199535,9cyclictest23-21ksoftirqd/110:09:421
1463501995314,6cyclictest131rcu_preempt11:40:021
1463501995311,5cyclictest131rcu_preempt11:30:211
1463496995317,5cyclictest1525476-21munin-run11:20:020
1463496995313,36cyclictest0-21swapper/008:53:190
146351099525,5cyclictest131rcu_preempt08:40:003
1463510995211,5cyclictest131rcu_preempt10:52:253
1463510995211,5cyclictest131rcu_preempt09:10:113
1463505995245,4cyclictest1538557-21kworker/2:011:58:312
1463505995244,5cyclictest1538557-21kworker/2:012:33:552
1463505995244,5cyclictest1538557-21kworker/2:012:28:572
1463505995244,5cyclictest1467398-21kworker/2:311:52:012
1463505995239,6cyclictest1467398-21kworker/2:311:12:002
1463505995238,6cyclictest1467398-21kworker/2:311:00:222
1463505995238,6cyclictest1467398-21kworker/2:310:07:202
1463505995234,12cyclictest3467602-21systemd-journal08:45:012
1463505995234,12cyclictest3467602-21systemd-journal08:45:012
1463505995211,8cyclictest131rcu_preempt11:45:122
1463505995210,5cyclictest131rcu_preempt09:17:012
1463505995210,5cyclictest131rcu_preempt09:17:002
146350199528,8cyclictest23-21ksoftirqd/112:11:081
146350199527,6cyclictest131rcu_preempt11:13:531
1463501995212,5cyclictest131rcu_preempt09:15:571
1463501995212,5cyclictest131rcu_preempt09:15:571
1463501995211,5cyclictest131rcu_preempt11:48:281
1463501995210,5cyclictest131rcu_preempt10:03:041
1463496995231,17cyclictest16850irq/134-800000011:05:380
1463496995225,12cyclictest1552120-21ssh12:22:170
1463496995222,27cyclictest19350irq/34-mmc109:40:000
1463496995222,27cyclictest19350irq/34-mmc109:40:000
146351099517,8cyclictest37-21ksoftirqd/307:45:003
146351099514,7cyclictest37-21ksoftirqd/310:21:083
1463510995111,7cyclictest131rcu_preempt12:13:113
146350599518,8cyclictest30-21ksoftirqd/210:50:012
146350599518,8cyclictest30-21ksoftirqd/210:50:012
146350599517,9cyclictest30-21ksoftirqd/209:56:302
146350599517,9cyclictest30-21ksoftirqd/209:56:302
1463505995144,4cyclictest1467398-21kworker/2:312:23:182
1463505995144,4cyclictest1463100-21kworker/2:008:16:512
1463505995143,5cyclictest1538557-21kworker/2:012:37:112
1463505995143,5cyclictest1467398-21kworker/2:312:07:232
1463505995143,5cyclictest1467398-21kworker/2:312:07:232
1463505995143,5cyclictest1467398-21kworker/2:310:56:282
1463505995141,6cyclictest1467398-21kworker/2:309:44:392
1463505995110,6cyclictest131rcu_preempt09:20:192
1463505995110,5cyclictest131rcu_preempt07:55:012
146350199518,5cyclictest131rcu_preempt12:20:581
1463501995131,15cyclictest275-21dbus-daemon08:00:001
1463501995117,6cyclictest131rcu_preempt09:45:491
1463501995117,6cyclictest131rcu_preempt09:45:481
1463501995111,5cyclictest131rcu_preempt12:05:581
1463501995111,5cyclictest131rcu_preempt12:05:571
1463501995111,5cyclictest131rcu_preempt10:30:481
1463501995111,11cyclictest23-21ksoftirqd/110:41:141
1463496995130,16cyclictest364-21snmpd08:31:400
1463496995130,16cyclictest364-21snmpd08:31:390
1463496995129,16cyclictest3467602-21systemd-journal07:40:000
1463496995129,16cyclictest3467602-21systemd-journal07:40:000
146351099509,4cyclictest131rcu_preempt08:25:013
146351099504,7cyclictest37-21ksoftirqd/309:00:023
146351099502,12cyclictest37-21ksoftirqd/308:50:013
1463505995042,5cyclictest1467398-21kworker/2:312:02:502
1463505995042,5cyclictest1467398-21kworker/2:311:41:172
1463505995041,6cyclictest1467398-21kworker/2:310:25:142
1463505995041,6cyclictest1467398-21kworker/2:310:25:142
146350199507,6cyclictest131rcu_preempt09:28:241
146350199506,8cyclictest23-21ksoftirqd/110:11:251
146350199506,8cyclictest23-21ksoftirqd/109:52:351
146350199505,10cyclictest23-21ksoftirqd/107:36:201
146350199505,10cyclictest23-21ksoftirqd/107:36:191
1463501995014,5cyclictest131rcu_preempt10:21:301
1463496995038,9cyclictest1464245-21sshd07:22:190
1463496995038,9cyclictest1464245-21sshd07:22:190
1463496995035,11cyclictest275-21dbus-daemon08:25:000
1463496995030,15cyclictest3467602-21systemd-journal09:00:000
1463496995029,15cyclictest1-21systemd08:50:010
1463496995019,14cyclictest1470745-21ssh09:12:090
146351099491,6cyclictest1463727-21modprobe07:15:003
1463505994942,4cyclictest1463100-21kworker/2:007:50:012
1463505994941,5cyclictest1467398-21kworker/2:312:15:252
1463505994941,5cyclictest1467398-21kworker/2:310:53:112
1463505994941,5cyclictest1467398-21kworker/2:310:23:122
1463505994941,5cyclictest1467398-21kworker/2:309:10:032
1463505994941,5cyclictest1463100-21kworker/2:008:51:502
1463505994937,5cyclictest1463100-21kworker/2:007:32:212
1463505994935,7cyclictest1467398-21kworker/2:309:01:592
1463505994926,9cyclictest30-21ksoftirqd/210:11:012
146350599491,9cyclictest30-21ksoftirqd/207:20:002
146350199498,5cyclictest131rcu_preempt10:27:161
146350199498,5cyclictest131rcu_preempt10:27:151
146350199496,5cyclictest131rcu_preempt09:05:011
1463501994912,6cyclictest131rcu_preempt12:33:081
146349699493,6cyclictest3467602-21systemd-journal11:47:360
1463496994922,20cyclictest275-21dbus-daemon07:20:010
1463496994914,30cyclictest1466729-21latency_hist08:10:010
146351099488,6cyclictest131rcu_preempt07:20:023
146351099487,5cyclictest131rcu_preempt07:55:193
1463510994810,12cyclictest37-21ksoftirqd/309:05:013
1463505994840,5cyclictest1463100-21kworker/2:008:36:022
1463505994840,5cyclictest1463100-21kworker/2:008:25:342
1463505994840,5cyclictest1463100-21kworker/2:008:14:122
1463501994812,6cyclictest131rcu_preempt07:45:001
146349699486,14cyclictest1465320-21systemd-run07:45:000
1463496994829,12cyclictest309-21in:imuxsock07:25:020
146351099477,7cyclictest37-21ksoftirqd/308:00:193
146351099477,7cyclictest37-21ksoftirqd/308:00:193
146351099475,8cyclictest37-21ksoftirqd/308:55:023
146351099472,8cyclictest37-21ksoftirqd/307:30:013
1463505994739,5cyclictest1463100-21kworker/2:008:04:202
1463505994739,5cyclictest1463100-21kworker/2:008:04:202
1463505994739,5cyclictest1463100-21kworker/2:007:22:492
1463505994739,5cyclictest1463100-21kworker/2:007:22:482
1463505994739,5cyclictest1463100-21kworker/2:007:14:142
1463505994738,6cyclictest1463100-21kworker/2:008:20:512
146350199477,7cyclictest23-21ksoftirqd/109:10:011
146350199477,7cyclictest23-21ksoftirqd/109:10:001
1463501994713,9cyclictest23-21ksoftirqd/107:14:101
1463501994711,9cyclictest23-21ksoftirqd/108:25:001
1463496994724,19cyclictest1463498-21latency_hist07:10:010
1463496994721,17cyclictest275-21dbus-daemon09:05:010
146351099461,11cyclictest37-21ksoftirqd/308:41:193
146351099461,11cyclictest37-21ksoftirqd/308:41:193
1463505994638,5cyclictest1463100-21kworker/2:008:47:232
1463505994638,5cyclictest1463100-21kworker/2:007:29:342
146350199465,8cyclictest23-21ksoftirqd/108:35:011
146350199465,8cyclictest23-21ksoftirqd/108:35:001
146350199463,8cyclictest23-21ksoftirqd/108:13:191
146350199462,11cyclictest23-21ksoftirqd/108:55:011
146350199459,5cyclictest131rcu_preempt08:25:191
1463501994537,5cyclictest23-21ksoftirqd/108:40:001
1463501994510,5cyclictest131rcu_preempt08:58:391
1463501994510,10cyclictest23-21ksoftirqd/108:07:361
1463501994510,10cyclictest23-21ksoftirqd/107:35:001
1463496994530,11cyclictest1465822-21/usr/sbin/munin07:55:020
1463496994510,13cyclictest1466474-21latency_hist08:05:020
1463496994510,13cyclictest1466474-21latency_hist08:05:020
146351099446,7cyclictest37-21ksoftirqd/307:55:003
1463505994435,6cyclictest1463100-21kworker/2:008:05:062
146350199444,9cyclictest23-21ksoftirqd/107:21:191
146350199444,9cyclictest23-21ksoftirqd/107:21:191
146349699444,31cyclictest1463735-21munin-plugin-st07:15:000
146350199439,5cyclictest131rcu_preempt08:41:541
146350199439,5cyclictest131rcu_preempt08:41:531
146350199431,6cyclictest0-21swapper/107:26:201
1463496994335,5cyclictest1467605-21sshd08:26:190
1463496994328,11cyclictest1467060-21sshd08:15:190
1463496994325,11cyclictest1468433-21systemd-run08:45:010
1463496994325,11cyclictest1468433-21systemd-run08:45:010
146351099423,11cyclictest37-21ksoftirqd/307:10:013
146350199426,8cyclictest23-21ksoftirqd/108:45:191
1463501994226,9cyclictest23-21ksoftirqd/108:15:531
146351099418,5cyclictest131rcu_preempt08:15:093
146351099416,9cyclictest37-21ksoftirqd/308:32:583
146351099416,9cyclictest37-21ksoftirqd/308:32:583
146350599341,5cyclictest131rcu_preempt07:10:012
1463501993021,6cyclictest23-21ksoftirqd/107:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional