You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 06:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun May 17, 2026 00:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13768149913720,7cyclictest131rcu_preempt00:10:013
1376802991146,5cyclictest131rcu_preempt23:40:021
1376810997419,31cyclictest1382976-21diskmemload23:21:452
137681099722,46cyclictest1394004-21ssh21:32:532
137681099682,61cyclictest3467602-21systemd-journal23:25:152
1376810996818,46cyclictest1471603-21sshd00:14:572
137681099663,42cyclictest1401424-21ssh21:48:122
1376810996626,36cyclictest1439875-21sh23:08:332
137681099662,19cyclictest1454640-21sh23:39:272
1376810996620,41cyclictest1385471-21cat21:15:022
1376810996620,41cyclictest1385471-21cat21:15:012
137681099661,60cyclictest1430403-21sshd22:48:582
137681099652,58cyclictest1433902-21ssh22:55:492
137681099652,58cyclictest1381387-21sshd20:38:572
137681099642,56cyclictest1472511-21ssh00:16:302
137681099642,31cyclictest1426865-21ssh22:41:082
137681099642,14cyclictest1380666-21munin-run20:25:012
1376810996421,40cyclictest30-21ksoftirqd/222:13:422
137681099641,59cyclictest1422769-21ssh22:33:022
1376810996335,23cyclictest1381158-21/usr/sbin/munin20:35:012
1376810996334,25cyclictest1-21systemd19:15:012
1376810996332,9cyclictest1382203-21kworker/2:323:50:022
137681099632,57cyclictest1466551-21ssh00:04:242
137681099632,57cyclictest1466551-21ssh00:04:232
137681099632,57cyclictest1396165-21ssh21:37:182
137681099631,9cyclictest1477277-21ssh00:26:212
137681099631,58cyclictest1402537-21ssh21:50:282
137681099631,13cyclictest48-21kcompactd022:20:192
137681099631,10cyclictest1421039-21ssh22:29:312
137681099631,10cyclictest1421039-21ssh22:29:312
1376802996320,12cyclictest23-21ksoftirqd/121:35:011
1376802996312,6cyclictest131rcu_preempt20:10:011
137681099623,55cyclictest1391193-21ssh21:26:492
137681099622,7cyclictest1474894-21ssh00:21:182
137681099622,7cyclictest1388652-21ssh21:21:282
137681099622,6cyclictest1386822-21ssh21:17:522
137681099622,54cyclictest1479364-21sh00:30:372
1376810996222,35cyclictest1405636-21sh21:57:002
1376810996222,35cyclictest1405636-21sh21:56:592
137681099621,8cyclictest1436370-21ssh23:00:572
137681099621,8cyclictest1432079-21ssh22:52:082
137681099621,8cyclictest1408490-21ssh22:03:112
1376810996218,40cyclictest30-21ksoftirqd/223:58:072
137681099621,5cyclictest1424862-21ssh22:37:092
137681099621,5cyclictest1424862-21ssh22:37:092
137681099621,57cyclictest1450330-21ssh23:30:082
137681099621,56cyclictest1468623-21ssh00:08:362
137681099621,15cyclictest1442625-21sh23:14:232
137681099621,15cyclictest1442625-21sh23:14:232
137681099621,15cyclictest1378330-21systemd-run19:40:012
1376799996239,16cyclictest3467602-21systemd-journal23:20:000
1376799996239,16cyclictest3467602-21systemd-journal23:20:000
137681099612,54cyclictest1381940-21munin-run20:50:012
137681099612,54cyclictest1377541-21munin-run19:25:002
137681099611,9cyclictest1460375-21ssh23:51:022
137681099611,9cyclictest1460375-21ssh23:51:012
137681099611,9cyclictest1443331-21ssh23:15:332
137681099611,9cyclictest1443331-21ssh23:15:322
137681099611,9cyclictest1410545-21ssh22:07:222
137681099611,9cyclictest1410545-21ssh22:07:222
137681099611,6cyclictest1456148-21ssh23:42:182
1376802996115,13cyclictest23-21ksoftirqd/100:05:011
1376802996115,13cyclictest23-21ksoftirqd/100:05:011
1376802996115,12cyclictest23-21ksoftirqd/100:00:021
1376799996128,29cyclictest1452533-21kworker/u8:300:05:020
1376799996128,29cyclictest1452533-21kworker/u8:300:05:020
137681099601,9cyclictest364-21snmpd20:06:132
137681099601,7cyclictest1415302-21ssh22:17:182
137681099601,7cyclictest1415302-21ssh22:17:172
137681099601,5cyclictest1398565-21ssh21:42:122
137681099601,55cyclictest1481692-21ssh00:35:282
1376802996015,10cyclictest23-21ksoftirqd/123:47:221
1376799996033,9cyclictest1454904-21modprobe23:40:000
1376799996022,11cyclictest12-21ksoftirqd/023:30:450
137681099592,52cyclictest1379337-21munin-run20:00:012
137681099591,5cyclictest1377272-21munin-run19:20:012
137681099591,5cyclictest1377272-21munin-run19:20:002
137681099591,53cyclictest309-21in:imuxsock20:42:552
137681099591,52cyclictest1380917-21munin-run20:30:022
137681099591,13cyclictest364-21snmpd19:26:442
1376802995920,8cyclictest23-21ksoftirqd/121:45:011
137681099583,48cyclictest275-21dbus-daemon20:55:012
137681099582,5cyclictest1378057-21munin-run19:35:012
137681099582,52cyclictest1380346-21sshd20:17:512
137681099582,52cyclictest1378827-21munin-run19:50:012
137681099582,52cyclictest1378827-21munin-run19:50:012
1376799995839,15cyclictest1-21systemd23:00:000
1376799995827,13cyclictest1438129-21ssh23:05:020
1376799995826,12cyclictest1398268-21ssh21:41:340
137681099572,51cyclictest364-21snmpd20:11:142
137681099572,50cyclictest1-21systemd21:05:012
137681099572,49cyclictest0-21swapper/219:41:072
137681099571,51cyclictest1379576-21sshd20:00:582
137680299578,7cyclictest23-21ksoftirqd/121:16:511
1376802995739,14cyclictest23-21ksoftirqd/122:50:011
1376802995714,11cyclictest23-21ksoftirqd/123:27:491
1376799995737,16cyclictest1-21systemd22:05:010
137681099561,51cyclictest364-21snmpd19:51:102
137680299567,7cyclictest23-21ksoftirqd/122:52:341
137680299566,7cyclictest23-21ksoftirqd/122:45:011
137680299563,8cyclictest23-21ksoftirqd/121:38:251
1376802995620,8cyclictest23-21ksoftirqd/100:10:001
1376799995628,13cyclictest1477948-21ssh00:27:560
1376799995628,12cyclictest1458126-21ssh23:46:280
1376799995627,14cyclictest1394894-21ssh21:34:560
1376799995626,10cyclictest1464549-21munin-run00:00:020
1376799995536,14cyclictest1-21systemd00:40:000
1376799995527,13cyclictest1388501-21ssh21:21:070
1376799995525,14cyclictest1401007-21ssh21:47:210
137680299548,6cyclictest23-21ksoftirqd/100:35:341
137680299547,8cyclictest23-21ksoftirqd/122:00:281
137680299545,12cyclictest23-21ksoftirqd/120:30:011
1376799995439,11cyclictest1-21systemd22:20:010
1376799995439,11cyclictest1-21systemd22:20:010
1376799995433,5cyclictest16850irq/134-800000000:10:010
1376799995427,12cyclictest1385594-21ssh21:15:080
1376799995426,13cyclictest1411414-21ssh22:09:190
1376799995426,13cyclictest1411414-21ssh22:09:190
1376799995425,4cyclictest131rcu_preempt22:10:410
1376799995418,6cyclictest16850irq/134-800000021:51:530
137680299537,12cyclictest23-21ksoftirqd/123:16:541
137680299537,12cyclictest23-21ksoftirqd/123:16:541
1376799995327,10cyclictest1419467-21ssh22:25:500
1376799995327,10cyclictest1419467-21ssh22:25:490
1376799995325,12cyclictest1423948-21ssh22:35:070
1376799995325,12cyclictest1423948-21ssh22:35:060
137681099521,46cyclictest1383212-21ssh21:10:032
137681099521,3cyclictest364-21snmpd20:56:232
137680299527,19cyclictest23-21ksoftirqd/123:40:571
137680299524,7cyclictest23-21ksoftirqd/100:15:181
1376802995235,12cyclictest1381420-21systemctl20:40:001
137679999525,26cyclictest12-21ksoftirqd/022:34:530
1376799995226,11cyclictest1429443-21ssh22:46:350
1376799995225,12cyclictest1446618-21ssh23:22:220
1376799995224,13cyclictest1471557-21ssh00:14:510
137681099511,15cyclictest1376882-21latency_hist19:10:012
137680299515,7cyclictest23-21ksoftirqd/123:55:031
137680299514,9cyclictest23-21ksoftirqd/100:20:111
1376802995138,9cyclictest23-21ksoftirqd/122:17:571
1376802995138,9cyclictest23-21ksoftirqd/122:17:571
137680299513,6cyclictest23-21ksoftirqd/119:45:021
1376802995113,14cyclictest23-21ksoftirqd/119:35:021
1376802995110,9cyclictest23-21ksoftirqd/123:03:151
1376799995142,6cyclictest16850irq/134-800000021:15:010
1376799995142,6cyclictest16850irq/134-800000021:15:010
137679999513,43cyclictest1379330-21latency_hist20:00:000
1376799995130,7cyclictest12-21ksoftirqd/022:44:240
1376799995127,9cyclictest1433255-21ssh22:54:550
1376799995124,12cyclictest1418328-21ssh22:23:350
1376814995037,10cyclictest0-21swapper/300:00:003
1376814995033,12cyclictest275-21dbus-daemon23:30:013
137680299508,6cyclictest131rcu_preempt21:59:011
137680299508,6cyclictest131rcu_preempt21:59:011
137680299507,7cyclictest23-21ksoftirqd/121:23:121
137680299506,11cyclictest23-21ksoftirqd/123:20:541
137680299505,9cyclictest23-21ksoftirqd/123:55:021
1376802995041,5cyclictest23-21ksoftirqd/119:30:011
137680299504,10cyclictest23-21ksoftirqd/121:47:271
1376802995010,4cyclictest131rcu_preempt22:27:401
1376802995010,4cyclictest131rcu_preempt22:27:391
137679999502,42cyclictest1462074-21munin-run23:55:010
137679999502,42cyclictest1462074-21munin-run23:55:010
1376799995023,11cyclictest1448890-21ssh23:27:110
1376799995023,11cyclictest1438627-21ssh23:05:390
137680299498,6cyclictest131rcu_preempt22:24:041
137680299497,9cyclictest23-21ksoftirqd/122:12:271
137680299496,8cyclictest23-21ksoftirqd/123:34:311
137680299495,12cyclictest23-21ksoftirqd/122:10:021
137680299495,12cyclictest23-21ksoftirqd/122:10:021
137680299494,8cyclictest23-21ksoftirqd/100:10:111
137680299494,11cyclictest23-21ksoftirqd/100:26:341
1376802994914,11cyclictest23-21ksoftirqd/123:14:081
1376802994914,11cyclictest23-21ksoftirqd/123:14:071
137680299491,13cyclictest23-21ksoftirqd/121:26:111
137679999497,26cyclictest1376795-21cyclictest20:25:020
1376799994941,5cyclictest16850irq/134-800000021:10:020
1376799994941,4cyclictest16850irq/134-800000000:16:580
1376799994922,12cyclictest1390611-21ssh21:25:360
137679999492,18cyclictest1442473-21ssh23:14:030
137679999492,18cyclictest1442473-21ssh23:14:020
1376814994834,10cyclictest3467602-21systemd-journal23:30:033
1376814994832,12cyclictest1-21systemd20:20:023
1376814994829,7cyclictest37-21ksoftirqd/323:40:023
137681499482,4cyclictest131rcu_preempt22:32:443
137680299486,10cyclictest23-21ksoftirqd/122:30:171
137680299484,9cyclictest23-21ksoftirqd/121:11:341
137680299484,9cyclictest23-21ksoftirqd/121:11:341
1376802994838,7cyclictest23-21ksoftirqd/122:59:421
1376802994831,13cyclictest1-21systemd19:55:011
1376802994815,11cyclictest23-21ksoftirqd/120:20:031
1376799994836,9cyclictest1392089-21kworker/u8:321:40:010
1376799994835,9cyclictest3467602-21systemd-journal20:20:010
137679999483,3cyclictest131rcu_preempt23:40:290
1376799994826,15cyclictest275-21dbus-daemon20:15:010
1376799994823,23cyclictest12-21ksoftirqd/000:22:160
1376814994729,12cyclictest275-21dbus-daemon19:45:013
137680299474,11cyclictest23-21ksoftirqd/122:36:551
137680299474,11cyclictest23-21ksoftirqd/122:36:541
1376802994721,17cyclictest221rcuc/123:09:251
1376802994711,6cyclictest131rcu_preempt20:04:571
1376799994731,12cyclictest1-21systemd20:40:010
1376799994726,7cyclictest12-21ksoftirqd/021:59:410
1376799994726,7cyclictest12-21ksoftirqd/021:59:400
1376814994626,14cyclictest275-21dbus-daemon23:20:013
1376814994626,14cyclictest275-21dbus-daemon23:20:013
137680299469,5cyclictest131rcu_preempt00:30:201
137680299464,8cyclictest23-21ksoftirqd/119:24:591
1376802994632,10cyclictest23-21ksoftirqd/120:45:011
137681499453,3cyclictest131rcu_preempt00:35:023
1376814994529,13cyclictest37-21ksoftirqd/321:40:003
137680299457,4cyclictest131rcu_preempt19:50:011
137680299457,4cyclictest131rcu_preempt19:50:011
137680299455,9cyclictest23-21ksoftirqd/121:53:461
1376802994533,9cyclictest23-21ksoftirqd/120:00:001
137680299453,10cyclictest23-21ksoftirqd/119:10:011
1376799994514,28cyclictest12-21ksoftirqd/000:32:320
137679999451,14cyclictest1381685-21munin-plugin-st20:45:000
1376814994435,5cyclictest1387807-21sendmail-msp21:20:003
1376814994431,11cyclictest37-21ksoftirqd/321:47:403
1376814994429,11cyclictest1-21systemd23:25:003
1376814994428,12cyclictest275-21dbus-daemon20:00:013
137681499441,39cyclictest37-21ksoftirqd/320:30:033
137680299441,9cyclictest1382189-21latency_hist20:55:001
137680299441,7cyclictest1381310-21sed20:35:021
137679999441,38cyclictest1378582-21true19:45:010
137681499434,12cyclictest0-21swapper/321:54:593
1376814994327,13cyclictest37-21ksoftirqd/320:25:013
1376814994324,15cyclictest37-21ksoftirqd/321:20:063
137681499432,3cyclictest131rcu_preempt22:21:543
137681499431,3cyclictest131rcu_preempt21:42:533
1376802994332,8cyclictest23-21ksoftirqd/119:39:591
1376799994336,4cyclictest16950irq/224-800000019:40:000
137681499426,25cyclictest1416684-21ssh22:20:023
137681499426,25cyclictest1416684-21ssh22:20:013
1376814994229,11cyclictest37-21ksoftirqd/322:06:583
1376814994229,11cyclictest37-21ksoftirqd/322:06:583
137681499421,36cyclictest1458296-21ssh23:46:493
137680299426,7cyclictest23-21ksoftirqd/120:15:021
1376814994126,11cyclictest309-21in:imuxsock20:00:033
1376814994126,11cyclictest275-21dbus-daemon22:00:013
1376814994126,11cyclictest275-21dbus-daemon22:00:013
1376814994123,14cyclictest275-21dbus-daemon22:50:003
1376814994123,13cyclictest275-21dbus-daemon20:45:013
137681499411,4cyclictest131rcu_preempt21:31:173
137680299417,7cyclictest23-21ksoftirqd/120:48:391
137681499405,3cyclictest131rcu_preempt22:39:563
137681499405,3cyclictest131rcu_preempt22:39:563
1376814994031,6cyclictest309-21in:imuxsock20:52:563
1376814994025,11cyclictest37-21ksoftirqd/323:02:083
1376814994024,12cyclictest37-21ksoftirqd/322:02:183
1376814994023,12cyclictest275-21dbus-daemon19:35:003
1376814994018,16cyclictest3467602-21systemd-journal19:14:593
137681499401,3cyclictest131rcu_preempt23:54:403
137681499401,3cyclictest131rcu_preempt23:54:393
137680299401,7cyclictest23-21ksoftirqd/119:10:571
1376799994031,6cyclictest1379063-21cron19:54:590
1376799994030,7cyclictest19350irq/34-mmc119:35:000
137679999402,34cyclictest1376951-21cat19:10:020
137681499394,9cyclictest1470712-1kworker/u9:000:23:303
1376814993929,8cyclictest37-21ksoftirqd/323:10:013
1376814993926,10cyclictest37-21ksoftirqd/322:42:063
1376814993923,10cyclictest1471831-21cat00:15:023
137681499391,3cyclictest131rcu_preempt21:14:423
137681499391,3cyclictest131rcu_preempt21:14:413
1376802993927,8cyclictest3467602-21systemd-journal19:20:011
1376802993927,8cyclictest3467602-21systemd-journal19:20:001
1376802993913,5cyclictest131rcu_preempt20:25:011
1376799993924,12cyclictest12-21ksoftirqd/019:24:380
137679999391,34cyclictest1382634-21latency_hist21:00:020
137681499389,3cyclictest131rcu_preempt19:39:593
137681499383,3cyclictest131rcu_preempt00:26:473
1376814993827,9cyclictest37-21ksoftirqd/322:55:073
1376814993827,9cyclictest37-21ksoftirqd/320:15:023
1376814993826,9cyclictest37-21ksoftirqd/322:52:193
1376814993826,10cyclictest37-21ksoftirqd/322:26:493
1376814993826,10cyclictest37-21ksoftirqd/322:26:483
1376814993825,9cyclictest309-21in:imuxsock19:23:573
1376814993825,9cyclictest275-21dbus-daemon20:40:003
137681499382,3cyclictest131rcu_preempt23:40:573
1376814993823,12cyclictest37-21ksoftirqd/323:11:413
1376814993823,12cyclictest37-21ksoftirqd/323:11:413
1376814993822,9cyclictest37-21ksoftirqd/320:50:003
1376814993822,11cyclictest37-21ksoftirqd/300:00:333
1376814993822,11cyclictest37-21ksoftirqd/300:00:323
137680299382,7cyclictest23-21ksoftirqd/121:05:011
137680299381,5cyclictest150-21pr/ttyS220:55:571
137680299381,5cyclictest0-21swapper/121:06:591
1376799993826,7cyclictest309-21in:imuxsock19:20:010
1376799993826,7cyclictest309-21in:imuxsock19:20:000
137679999382,5cyclictest0-21swapper/019:15:010
137679999381,4cyclictest0-21swapper/021:05:000
137681499372,3cyclictest131rcu_preempt21:27:323
1376814993723,11cyclictest37-21ksoftirqd/322:10:253
137681499371,3cyclictest131rcu_preempt00:35:113
1376799993727,6cyclictest12-21ksoftirqd/019:50:030
1376799993727,6cyclictest12-21ksoftirqd/019:50:030
1376799993725,8cyclictest12-21ksoftirqd/019:29:290
137679999371,32cyclictest1379578-21sshd20:00:580
137681499365,4cyclictest131rcu_preempt00:20:003
1376814993623,10cyclictest3467602-21systemd-journal21:07:583
1376814993622,10cyclictest37-21ksoftirqd/320:35:023
137679999363,6cyclictest3467602-21systemd-journal20:09:570
1376799993623,9cyclictest12-21ksoftirqd/020:52:000
1376799993621,11cyclictest1381127-21sshd20:31:570
1376799993617,15cyclictest275-21dbus-daemon20:30:010
137679999361,32cyclictest0-21swapper/020:46:470
1376814993523,9cyclictest37-21ksoftirqd/319:50:023
1376814993523,9cyclictest37-21ksoftirqd/319:50:013
137681499345,3cyclictest131rcu_preempt20:10:003
1376814993421,11cyclictest37-21ksoftirqd/319:55:003
1376814993322,9cyclictest37-21ksoftirqd/321:05:013
1376814993221,5cyclictest1377791-21munin-plugin-st19:30:003
1376814993018,9cyclictest309-21in:imuxsock19:15:033
1376814993018,9cyclictest309-21in:imuxsock19:15:033
137681499291,3cyclictest131rcu_preempt19:10:023
1376814992711,12cyclictest275-21dbus-daemon21:00:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional