You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 19:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 25, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1426279998728,15cyclictest1482686-21ssh10:55:120
1426279998527,11cyclictest1480384-21ssh10:50:260
1426279998229,14cyclictest1456237-21sed10:00:010
1426279998025,14cyclictest1443616-21ssh09:33:260
1426279997929,12cyclictest1471291-21ssh10:31:310
1426279997825,13cyclictest1510222-21ssh11:53:200
1426279997824,12cyclictest1523503-21ssh12:20:580
1426279997814,59cyclictest16850irq/134-800000009:51:460
1426283997724,17cyclictest1-21systemd10:40:011
1426279997719,53cyclictest16850irq/134-800000012:14:370
1426279997613,18cyclictest16850irq/134-800000009:28:230
1426287997532,38cyclictest1-21systemd11:30:002
1426279997529,13cyclictest1528395-21ssh12:31:190
1426279997529,12cyclictest1511981-21ssh11:56:500
1426279997515,20cyclictest16850irq/134-800000009:48:030
1426279997515,20cyclictest16850irq/134-800000009:48:020
1426279997427,14cyclictest1448777-21ssh09:44:200
1426279997424,15cyclictest1435046-21ssh09:15:100
142627999742,37cyclictest1495773-21ssh11:22:560
1426279997418,51cyclictest16850irq/134-800000012:19:010
1426279997329,4cyclictest16950irq/224-800000007:34:130
1426279997319,23cyclictest16850irq/134-800000010:19:470
1426279997318,49cyclictest16850irq/134-800000011:49:280
1426279997236,32cyclictest16950irq/224-800000009:12:110
1426279997217,49cyclictest16850irq/134-800000010:07:590
1426279997217,49cyclictest16850irq/134-800000010:07:590
1426283997131,11cyclictest23-21ksoftirqd/112:25:011
142627999716,34cyclictest16850irq/134-800000010:02:200
1426279997126,11cyclictest1499731-21ssh11:30:580
1426279997118,49cyclictest16850irq/134-800000012:28:010
1426279997116,50cyclictest16850irq/134-800000010:25:220
1426279997116,50cyclictest16850irq/134-800000010:25:210
1426279997112,20cyclictest16850irq/134-800000010:44:410
142628799702,14cyclictest1427487-21sshd07:31:102
1426279997018,46cyclictest16850irq/134-800000009:20:340
1426279997016,49cyclictest16850irq/134-800000011:44:410
1426289996929,35cyclictest309-21in:imuxsock08:05:003
1426287996923,42cyclictest30-21ksoftirqd/208:34:112
1426279996929,13cyclictest1518128-21latency_hist12:10:010
1426289996817,14cyclictest1-21systemd09:10:013
142627999683,28cyclictest149792850ssh11:27:220
1426287996722,41cyclictest1496091-21sh11:23:462
142628799671,13cyclictest1440257-21ssh09:26:132
142627999673,6cyclictest16850irq/134-800000009:39:280
142627999672,6cyclictest1532423-21modprobe12:40:010
142627999672,6cyclictest1532423-21modprobe12:40:010
1426279996711,50cyclictest16850irq/134-800000011:14:480
1426279996711,50cyclictest16850irq/134-800000011:14:480
1426289996626,36cyclictest1431922-21cat08:59:593
1426289996626,36cyclictest1-21systemd07:40:003
1426287996621,40cyclictest1428946-21cut08:00:022
142628799661,54cyclictest1432452-21diskmemload11:15:022
142628799661,54cyclictest1432452-21diskmemload11:15:012
142628799661,13cyclictest1439194-21sshd09:24:112
1426279996629,12cyclictest1485392-21ssh11:00:540
1426279996615,46cyclictest16850irq/134-800000011:16:240
1426279996613,48cyclictest16850irq/134-800000010:24:580
1426279996525,11cyclictest1514479-21sshd12:02:090
142627999651,25cyclictest1489170-21ssh11:09:110
1426289996427,31cyclictest1477553-21latency_hist10:45:013
1426289996423,12cyclictest37-21ksoftirqd/311:14:453
1426289996423,12cyclictest37-21ksoftirqd/311:14:443
142628799642,13cyclictest1468099-21ssh10:25:002
142628799642,12cyclictest1472423-21ssh10:34:122
142627999649,7cyclictest16850irq/134-800000011:39:520
142627999648,51cyclictest16850irq/134-800000010:47:440
1426279996415,44cyclictest16850irq/134-800000010:38:310
1426289996321,12cyclictest275-21dbus-daemon09:20:003
1426287996310,7cyclictest1427229-21sshd07:25:122
1426279996332,12cyclictest1430831-21sed08:35:030
1426289996216,10cyclictest1532570-21grep12:40:023
1426289996216,10cyclictest1532570-21grep12:40:023
1426287996227,30cyclictest1462814-21sh10:13:512
142628799622,17cyclictest1457207-21ssh10:01:522
1426287996220,37cyclictest1510936-21sh11:54:592
142628799621,14cyclictest1436570-21ssh09:18:452
142628799621,13cyclictest364-21snmpd09:54:112
1426289996123,5cyclictest1487126-21kworker/u8:411:39:593
1426289996121,5cyclictest1471898-21sh10:32:573
142628799612,14cyclictest1434583-21ssh09:14:322
142628799611,14cyclictest1531191-21sshd12:37:102
142628799611,14cyclictest1531191-21sshd12:37:092
142628799611,11cyclictest1473322-21ssh10:35:442
142627999619,18cyclictest275-21dbus-daemon07:40:010
142627999612,34cyclictest1432543-21ssh09:10:020
142627999612,32cyclictest16850irq/134-800000010:10:040
1426289996026,30cyclictest1521844-21sh12:17:433
1426289996023,8cyclictest37-21ksoftirqd/311:46:173
1426289996019,7cyclictest37-21ksoftirqd/310:46:413
142628799601,16cyclictest1478717-21ssh10:47:072
142628799601,15cyclictest1501303-21ssh11:34:482
142628799601,15cyclictest1449799-21sshd09:46:112
142628799601,15cyclictest1449799-21sshd09:46:102
142628799601,11cyclictest1493689-21ssh11:18:452
1426289995923,4cyclictest1465925-21latency_hist10:20:023
142628799592,16cyclictest1458951-21ssh10:05:312
142628799592,16cyclictest1458951-21ssh10:05:302
142628799591,14cyclictest364-21snmpd11:38:052
1426279995928,5cyclictest309-21in:imuxsock07:55:020
1426289995829,25cyclictest37-21ksoftirqd/309:25:403
1426289995825,29cyclictest1443272-21ssh09:32:443
1426289995821,6cyclictest37-21ksoftirqd/312:21:123
1426289995818,6cyclictest0-21swapper/311:05:393
142628799583,17cyclictest1518130-21latency_hist12:10:022
142628799582,17cyclictest1476214-21ssh10:41:512
142628799582,13cyclictest1426469-21munin-run07:15:002
1426287995814,5cyclictest1522884-21latency_hist12:20:012
1426287995814,12cyclictest30-21ksoftirqd/211:06:512
1426289995724,29cyclictest275-21dbus-daemon11:30:023
142628999572,15cyclictest1458727-21latency_hist10:05:013
1426289995718,6cyclictest1463293-21kworker/u8:110:23:343
142628799572,16cyclictest1448231-21ssh09:43:102
142628799572,15cyclictest1465702-21/usr/sbin/munin10:20:012
142628799572,14cyclictest1429821-21sshd08:17:102
142628799572,13cyclictest1480816-21ssh10:51:302
1426287995721,32cyclictest1505714-21sh11:43:572
1426287995718,34cyclictest275-21dbus-daemon07:45:002
1426287995714,37cyclictest1445888-21rm09:38:102
142628799571,13cyclictest364-21snmpd09:59:382
142628799571,12cyclictest364-21snmpd12:33:402
142628799571,11cyclictest364-21snmpd12:26:212
1426279995719,7cyclictest12-21ksoftirqd/008:55:010
1426279995719,7cyclictest12-21ksoftirqd/008:55:010
1426289995624,5cyclictest1455560-21sh09:58:353
1426289995621,6cyclictest37-21ksoftirqd/311:55:093
1426289995619,4cyclictest1432452-21diskmemload12:07:033
1426289995617,6cyclictest1427512-21munin-plugin-st07:35:003
142628799562,19cyclictest1518496-21ssh12:10:272
142628799562,11cyclictest1442545-21ssh09:31:032
142628799561,16cyclictest1487134-21/usr/sbin/munin11:05:002
142628799561,13cyclictest1482660-21modprobe10:55:102
1426289995523,28cyclictest1530068-21latency_hist12:35:033
1426289995518,8cyclictest37-21ksoftirqd/311:53:073
142628799553,16cyclictest1524439-21ssh12:23:042
142628799552,11cyclictest1514178-21ssh12:01:212
142628799551,13cyclictest364-21snmpd09:02:392
1426283995512,8cyclictest0-21swapper/110:25:011
1426289995419,12cyclictest37-21ksoftirqd/309:11:163
142628799542,13cyclictest1428565-21munin-run07:55:022
1426287995416,32cyclictest275-21dbus-daemon08:15:012
142628799541,13cyclictest364-21snmpd08:36:082
142628799541,13cyclictest1469539-21ssh10:28:012
142628799541,13cyclictest1469539-21ssh10:28:012
142628799541,11cyclictest1507076-21ssh11:46:312
142628399548,7cyclictest131rcu_preempt09:20:011
1426289995319,9cyclictest37-21ksoftirqd/310:54:583
1426289995313,11cyclictest0-21swapper/311:31:023
142628799532,12cyclictest1431640-21sshd08:52:102
142628799532,12cyclictest1431640-21sshd08:52:102
142628799532,11cyclictest1426771-21systemd-run07:20:012
1426287995315,33cyclictest1511700-21sshd11:56:152
142628999523,18cyclictest1429334-21munin-plugin-st08:10:003
1426289995222,7cyclictest37-21ksoftirqd/309:25:023
1426289995221,5cyclictest1432452-21diskmemload10:06:273
1426289995221,5cyclictest1432452-21diskmemload10:06:263
1426289995219,9cyclictest37-21ksoftirqd/309:42:113
1426289995217,7cyclictest1453998-21latency_hist09:55:033
142628799522,9cyclictest1432435-21cron09:10:012
1426283995234,13cyclictest275-21dbus-daemon09:30:001
1426289995119,9cyclictest37-21ksoftirqd/312:00:473
1426289995024,10cyclictest37-21ksoftirqd/310:12:023
1426289995018,6cyclictest309-21in:imuxsock10:40:013
1426289995016,25cyclictest0-21swapper/311:44:143
1426289995011,5cyclictest131rcu_preempt07:20:013
1426287995023,24cyclictest30-21ksoftirqd/207:25:022
142628799502,14cyclictest364-21snmpd08:27:552
1426287995013,9cyclictest1426362-21latency_hist07:10:012
1426287995010,4cyclictest0-21swapper/208:22:532
142628399502,4cyclictest131rcu_preempt12:40:031
142628399502,4cyclictest131rcu_preempt12:40:021
142628399502,4cyclictest131rcu_preempt11:55:071
142628399502,3cyclictest131rcu_preempt10:43:261
1426289994919,5cyclictest37-21ksoftirqd/309:39:293
1426289994919,4cyclictest1482854-21sh10:55:423
1426289994919,10cyclictest1426405-21date07:10:023
1426289994914,9cyclictest1494918-21sh11:20:553
1426287994941,5cyclictest1425874-21kworker/2:007:50:012
1426287994913,30cyclictest1427972-21grep07:40:032
142628399494,4cyclictest131rcu_preempt12:11:361
142628399493,6cyclictest1501802-21ssh11:35:211
1426279994940,5cyclictest16950irq/224-800000008:30:000
142628799481,13cyclictest1431952-21true09:00:012
142628399484,4cyclictest131rcu_preempt11:00:541
142628399484,4cyclictest131rcu_preempt10:50:031
142628399483,4cyclictest131rcu_preempt11:12:131
142628399483,4cyclictest131rcu_preempt11:12:121
142628399483,3cyclictest131rcu_preempt09:35:561
142628399483,3cyclictest131rcu_preempt07:20:021
1426283994833,10cyclictest1-21systemd08:25:011
142628399482,7cyclictest131rcu_preempt11:20:311
142628399482,5cyclictest131rcu_preempt11:17:031
142628399481,4cyclictest131rcu_preempt10:28:211
142628399481,4cyclictest131rcu_preempt10:28:201
142627999481,12cyclictest1426740-21grep07:20:000
1426289994720,9cyclictest37-21ksoftirqd/307:50:023
1426289994718,9cyclictest37-21ksoftirqd/310:29:563
1426289994718,9cyclictest37-21ksoftirqd/310:29:553
1426287994713,15cyclictest275-21dbus-daemon08:50:012
142628799471,12cyclictest1429212-21sed08:05:012
142628399474,4cyclictest131rcu_preempt11:34:421
142628399474,3cyclictest131rcu_preempt10:17:081
142628399473,4cyclictest131rcu_preempt10:53:321
1426283994734,9cyclictest1-21systemd11:00:011
142628399473,3cyclictest131rcu_preempt12:08:561
1426283994732,12cyclictest3467602-21systemd-journal08:51:111
1426283994732,12cyclictest3467602-21systemd-journal08:51:111
142628399472,5cyclictest131rcu_preempt09:42:441
142628399472,4cyclictest131rcu_preempt11:42:531
142628399472,3cyclictest131rcu_preempt12:15:051
142628399472,3cyclictest131rcu_preempt10:05:521
142628399472,3cyclictest131rcu_preempt10:05:521
142628399471,4cyclictest131rcu_preempt09:53:381
1426279994740,4cyclictest16950irq/224-800000008:40:000
1426289994629,13cyclictest275-21dbus-daemon07:30:013
142628999462,13cyclictest1427075-21sort07:25:013
142628399463,4cyclictest131rcu_preempt09:49:561
142628399463,4cyclictest131rcu_preempt09:49:551
142628399463,3cyclictest131rcu_preempt07:30:011
1426283994629,13cyclictest275-21dbus-daemon11:10:011
142628399462,3cyclictest131rcu_preempt11:46:111
142628399461,4cyclictest131rcu_preempt12:30:151
1426283994610,5cyclictest131rcu_preempt09:22:551
142627999462,38cyclictest1431925-21munin-plugin-st09:00:000
142628999455,8cyclictest37-21ksoftirqd/308:21:573
142628399454,4cyclictest131rcu_preempt10:04:101
1426283994532,9cyclictest3467602-21systemd-journal08:30:001
142628399452,4cyclictest131rcu_preempt11:54:011
142628399452,4cyclictest131rcu_preempt09:56:521
142628399452,4cyclictest131rcu_preempt07:50:021
142628399451,3cyclictest131rcu_preempt10:35:001
142628399451,15cyclictest1433757-21ssh09:12:401
142627999459,5cyclictest0-21swapper/007:45:020
1426279994529,11cyclictest275-21dbus-daemon08:25:020
142628999446,10cyclictest37-21ksoftirqd/307:58:133
142628999445,5cyclictest131rcu_preempt09:49:333
142628999445,5cyclictest131rcu_preempt09:49:333
1426283994435,5cyclictest1427008-21munin-plugin-st07:25:001
1426283994432,9cyclictest1-21systemd07:40:001
142628399442,7cyclictest1442723-21ssh09:31:261
1426283994427,13cyclictest1-21systemd08:50:001
142628399442,4cyclictest131rcu_preempt12:26:301
142628399442,3cyclictest131rcu_preempt08:38:101
142628399441,3cyclictest131rcu_preempt11:28:231
142628399441,3cyclictest131rcu_preempt09:06:091
142627999443,7cyclictest131rcu_preempt08:10:010
1426279994437,4cyclictest16950irq/224-800000008:50:020
142628999439,4cyclictest131rcu_preempt12:25:103
1426289994329,10cyclictest3467602-21systemd-journal08:40:023
142628799432,10cyclictest1-21systemd08:10:002
142628399434,4cyclictest131rcu_preempt08:15:021
142628399432,4cyclictest131rcu_preempt12:03:011
1426279994315,24cyclictest1428590-21latency_hist07:55:020
142628999421,4cyclictest131rcu_preempt11:19:593
142628999421,3cyclictest131rcu_preempt11:05:023
142628399421,5cyclictest0-21swapper/108:35:011
1426279994215,18cyclictest16850irq/134-800000008:10:350
142628399413,4cyclictest131rcu_preempt07:57:491
1426283994129,9cyclictest309-21in:imuxsock07:44:101
1426279994134,4cyclictest16950irq/224-800000008:40:420
142627999412,31cyclictest1426282-21latency_hist07:10:010
1426289994027,9cyclictest3467602-21systemd-journal08:30:013
1426289994027,10cyclictest37-21ksoftirqd/312:12:143
1426287994010,6cyclictest30-21ksoftirqd/208:45:012
142628399402,4cyclictest131rcu_preempt07:15:011
1426283994015,6cyclictest131rcu_preempt10:14:371
1426279994032,5cyclictest19350irq/34-mmc107:26:080
1426279994029,8cyclictest1426981-21sshd07:21:100
1426279994024,12cyclictest12-21ksoftirqd/007:15:010
142628999397,3cyclictest131rcu_preempt09:00:033
1426289993926,10cyclictest37-21ksoftirqd/308:20:003
142628399394,3cyclictest131rcu_preempt07:52:471
1426283993929,7cyclictest23-21ksoftirqd/109:00:021
142627999392,33cyclictest3467602-21systemd-journal09:02:010
142628999384,4cyclictest131rcu_preempt08:45:013
1426289993827,8cyclictest1-21systemd08:15:013
1426283993826,8cyclictest23-21ksoftirqd/108:45:021
142627999381,4cyclictest1429846-21modprobe08:20:000
1426283993729,5cyclictest1429332-21munin-plugin-st08:10:001
1426279993729,5cyclictest12-21ksoftirqd/007:50:020
142628999363,20cyclictest3467602-21systemd-journal07:55:003
1426283993617,6cyclictest23-21ksoftirqd/108:05:001
142628399361,4cyclictest131rcu_preempt07:30:521
142627999361,30cyclictest3467602-21systemd-journal08:04:340
1426289993524,8cyclictest1431503-21sed08:50:013
1426289993425,6cyclictest1431728-21latency_hist08:55:033
1426289993425,6cyclictest1431728-21latency_hist08:55:023
142628399342,29cyclictest1429842-21modprobe08:20:001
142628399341,3cyclictest131rcu_preempt09:01:111
1426289993324,6cyclictest310-21in:imklog07:42:513
142628999311,3cyclictest131rcu_preempt07:15:013
1426283993021,6cyclictest23-21ksoftirqd/107:10:011
1426289992917,10cyclictest37-21ksoftirqd/308:30:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional