You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-12 - 20:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 12, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1488763999462,27cyclictest23-21ksoftirqd/107:55:031
1488760998774,3cyclictest0-21swapper/007:45:010
1488760998320,45cyclictest1526512-21ssh10:20:120
1488760998228,36cyclictest1509877-21ssh09:43:310
1488760998226,52cyclictest1568461-21ssh11:54:170
1488760998226,52cyclictest1568461-21ssh11:54:160
1488760998030,45cyclictest1572341-21ssh12:02:440
1488760998030,45cyclictest1572341-21ssh12:02:440
148876099792,5cyclictest0-21swapper/012:25:020
1488760997829,45cyclictest16950irq/224-800000010:41:560
1488760997828,46cyclictest1556219-21ssh11:26:400
1488760997817,56cyclictest12-21ksoftirqd/012:31:230
1488760997667,3cyclictest0-21swapper/008:25:020
1488760997629,43cyclictest1514232-21ssh09:53:160
1488760997628,44cyclictest1499929-21ssh09:20:560
148876399751,70cyclictest1516875-21sshd09:59:251
148876099758,43cyclictest16850irq/134-800000011:23:580
148876099751,6cyclictest1505744-21ssh09:34:180
148876099736,57cyclictest16850irq/134-800000010:53:380
148876099722,6cyclictest1528744-21ssh10:25:100
1488760997213,49cyclictest16850irq/134-800000011:55:230
1488760997211,51cyclictest16850irq/134-800000009:19:510
1488760997210,58cyclictest16850irq/134-800000010:08:430
148876099716,13cyclictest1565902-21ssh11:48:320
148876099712,64cyclictest1553095-21sendmail-msp11:20:010
148876099701,8cyclictest1517151-21modprobe10:00:010
1488760997016,49cyclictest16850irq/134-800000011:32:340
1488760997014,50cyclictest16850irq/134-800000010:18:260
148876099683,5cyclictest1550170-21ssh11:13:220
1488760996812,51cyclictest16850irq/134-800000011:00:430
1488760996731,32cyclictest1494774-21sed09:05:020
1488760996725,37cyclictest309-21in:imuxsock12:15:010
1488760996725,37cyclictest309-21in:imuxsock12:15:010
148876099672,4cyclictest1497022-21ssh09:14:500
1488760996712,50cyclictest16850irq/134-800000009:38:140
148876099671,16cyclictest1534852-21sshd10:39:250
1488770996622,7cyclictest30-21ksoftirqd/211:00:032
1488770996618,9cyclictest30-21ksoftirqd/207:20:012
148876099667,54cyclictest16850irq/134-800000010:59:230
1488760996631,6cyclictest12-21ksoftirqd/009:50:010
148876099657,5cyclictest1493354-21sshd08:39:250
1488770996445,8cyclictest30-21ksoftirqd/211:40:012
148876399645,15cyclictest1519405-21modprobe10:05:001
1488763996421,39cyclictest275-21dbus-daemon11:20:011
148876099648,52cyclictest1587166-21ssh12:35:360
148876099641,7cyclictest3467602-21systemd-journal10:30:220
1488763996337,21cyclictest3467602-21systemd-journal12:13:251
1488763996337,21cyclictest3467602-21systemd-journal12:13:251
1488763996330,29cyclictest1521495-21sh10:09:401
148876399632,12cyclictest48-21kcompactd010:24:491
148876399631,15cyclictest1524759-21ssh10:16:321
148876099633,5cyclictest1519429-21munin-run10:05:000
1488770996223,27cyclictest1488755-21cyclictest11:30:032
1488763996220,37cyclictest1498595-21sh09:18:121
1488763996214,12cyclictest275-21dbus-daemon10:35:011
148876099629,36cyclictest16850irq/134-800000010:14:470
1488760996221,37cyclictest16950irq/224-800000008:30:010
1488760996218,40cyclictest1560079-21ssh11:35:120
148876099621,6cyclictest1548563-21sshd11:09:540
148876099621,6cyclictest1548563-21sshd11:09:530
1488770996116,8cyclictest30-21ksoftirqd/211:45:012
148876399612,9cyclictest1491282-21cron08:00:001
1488763996122,35cyclictest1572151-21sh12:02:181
1488763996122,35cyclictest1572151-21sh12:02:181
148876399612,11cyclictest1492056-21sshd08:14:141
148876399611,15cyclictest1504307-21ssh09:30:391
148876399611,13cyclictest1569771-21ssh11:57:041
148876399611,13cyclictest1533813-21ssh10:36:461
148876099612,13cyclictest1493663-21munin-run08:45:020
148876099612,13cyclictest1493663-21munin-run08:45:010
148876099611,6cyclictest1538174-21sshd10:46:250
148876099611,6cyclictest1503809-21munin-run09:30:010
148876099611,12cyclictest48-21kcompactd011:40:340
1488774996039,15cyclictest3467602-21systemd-journal11:25:033
148876399602,14cyclictest1538591-21ssh10:47:201
148876399602,13cyclictest1562502-21ssh11:40:381
148876399602,13cyclictest1514201-21ssh09:53:131
148876399601,16cyclictest1575392-21ssh12:09:471
148876399601,14cyclictest1567471-21ssh11:51:531
148876399601,14cyclictest1567471-21ssh11:51:531
148876399601,14cyclictest1556768-21ssh11:28:031
148876399601,13cyclictest1564776-21ssh11:45:381
148876399601,13cyclictest1561361-21ssh11:38:191
148876099602,4cyclictest1582785-21ssh12:25:510
1488760996020,35cyclictest1578111-21sh12:15:240
1488760996020,35cyclictest1578111-21sh12:15:230
148876399592,14cyclictest1578888-21ssh12:17:181
148876399592,14cyclictest1578888-21ssh12:17:181
148876399592,13cyclictest1587078-21ssh12:35:241
148876399592,13cyclictest1544696-21ssh11:00:501
148876399592,13cyclictest1536159-21ssh10:42:021
148876399592,13cyclictest1522779-21ssh10:12:081
1488763995921,32cyclictest309-21in:imuxsock09:10:001
148876399592,11cyclictest1547483-21ssh11:07:111
148876399592,11cyclictest1547483-21ssh11:07:111
148876399591,16cyclictest1507958-21sh09:39:181
148876399591,14cyclictest1586013-21ssh12:33:131
148876399591,14cyclictest1582747-21ssh12:25:451
148876399591,14cyclictest1509947-21ssh09:43:401
148876399591,13cyclictest1529884-21ssh10:28:041
148876399591,13cyclictest1501864-21ssh09:25:151
148876099592,15cyclictest1491040-21latency_hist07:55:010
148876099591,9cyclictest3467602-21systemd-journal12:07:240
1488770995815,9cyclictest30-21ksoftirqd/210:41:242
148876399582,15cyclictest1492505-21latency_hist08:20:031
148876399582,15cyclictest1492505-21latency_hist08:20:031
148876399582,14cyclictest1496249-21ssh09:13:011
148876399582,14cyclictest1494367-21sshd08:55:251
148876399582,14cyclictest1494367-21sshd08:55:251
148876399582,14cyclictest1491595-21grep08:05:011
148876399582,12cyclictest1512352-21ssh09:49:041
1488763995815,39cyclictest1489533-21latency_hist07:25:011
1488763995815,39cyclictest1489533-21latency_hist07:25:011
148876399581,14cyclictest1549152-21ssh11:10:501
148876399581,14cyclictest1542763-21ssh10:56:431
148876099582,9cyclictest1-21systemd09:00:010
148876099582,9cyclictest1-21systemd09:00:000
148876099582,5cyclictest364-21snmpd07:45:240
1488774995735,17cyclictest275-21dbus-daemon07:40:013
1488770995711,10cyclictest30-21ksoftirqd/212:18:062
1488770995711,10cyclictest30-21ksoftirqd/212:18:062
148876399571,13cyclictest1558385-21ssh11:31:321
148876399571,13cyclictest1553660-21ssh11:20:581
148876399571,13cyclictest1540509-21ssh10:51:381
148876399571,13cyclictest1501370-21ssh09:24:401
1488760995748,5cyclictest16950irq/224-800000007:40:010
148876099572,5cyclictest1489925-21latency_hist07:30:010
148876099572,17cyclictest1491296-21latency_hist07:59:590
148876099571,11cyclictest1491540-21sshd08:00:250
1488774995632,16cyclictest275-21dbus-daemon12:25:013
148877099564,17cyclictest1539000-21ssh10:48:242
1488770995616,8cyclictest30-21ksoftirqd/211:18:392
1488770995613,6cyclictest30-21ksoftirqd/209:50:142
1488770995610,9cyclictest1492240-21latency_hist08:15:022
1488770995610,18cyclictest1568860-21cut11:55:022
1488770995610,18cyclictest1568860-21cut11:55:012
148876399562,12cyclictest1581499-21ssh12:23:101
148876399562,12cyclictest1492856-21cron08:30:001
148876099562,8cyclictest1492078-21munin-run08:15:010
148876099561,2cyclictest0-21swapper/008:15:030
148876099561,2cyclictest0-21swapper/008:15:020
1488774995530,18cyclictest1553073-21sh11:20:013
1488774995525,20cyclictest275-21dbus-daemon08:45:013
1488774995525,20cyclictest275-21dbus-daemon08:45:003
148877099557,9cyclictest30-21ksoftirqd/211:05:012
148877099555,14cyclictest1520785-21ssh10:07:532
1488770995527,8cyclictest30-21ksoftirqd/211:22:052
148876399553,11cyclictest3467602-21systemd-journal07:20:001
148876399551,15cyclictest1488975-21unin-run07:15:011
148876099553,7cyclictest1489467-21sshd07:20:260
148876099553,7cyclictest1489467-21sshd07:20:260
148876099552,9cyclictest1494967-21ssh09:10:010
148876099552,10cyclictest1488948-21cron07:15:000
1488774995426,17cyclictest275-21dbus-daemon10:00:023
1488770995421,21cyclictest1492300-21kworker/u8:008:25:022
148876399541,12cyclictest364-21snmpd07:49:061
148876399541,12cyclictest1491814-21cron08:10:001
148876099543,13cyclictest1494140-21cron08:55:000
148876399532,13cyclictest1490276-21sendmail07:40:001
148876099532,7cyclictest16950irq/224-800000007:30:250
1488774995236,11cyclictest1-21systemd08:50:013
148877499521,32cyclictest361rcuc/312:30:043
148877099526,8cyclictest30-21ksoftirqd/212:11:342
148877099526,8cyclictest30-21ksoftirqd/212:11:342
1488770995214,7cyclictest30-21ksoftirqd/209:31:312
148876399524,9cyclictest1-21systemd08:40:011
148876399523,7cyclictest1492619-21grep08:24:591
148876399523,7cyclictest1489756-21modprobe07:30:001
148876399522,11cyclictest1494677-21modprobe09:04:591
148877099515,6cyclictest30-21ksoftirqd/210:17:222
1488770995114,9cyclictest30-21ksoftirqd/211:32:252
1488763995122,25cyclictest1-21systemd07:35:001
148876399511,12cyclictest364-21snmpd08:33:071
148876399511,12cyclictest364-21snmpd08:33:061
148876399511,12cyclictest1494126-21sshd08:52:251
148876399511,12cyclictest1493735-21latency_hist08:45:011
148876399511,12cyclictest1493735-21latency_hist08:45:011
148876099512,4cyclictest48-21kcompactd007:15:200
148876099511,6cyclictest364-21snmpd08:49:330
148877499505,31cyclictest1570947-21modprobe12:00:003
148877099508,11cyclictest30-21ksoftirqd/212:06:572
148877099506,8cyclictest30-21ksoftirqd/209:26:312
1488770995035,11cyclictest275-21dbus-daemon09:10:002
1488770995012,7cyclictest30-21ksoftirqd/211:10:032
1488770995012,7cyclictest30-21ksoftirqd/211:10:022
148876399502,10cyclictest1490598-21cat07:45:021
148877499491,3cyclictest131rcu_preempt10:25:293
148877099498,8cyclictest0-21swapper/210:24:202
1488770994939,5cyclictest1519415-21munin-plugin-st10:04:592
148877099493,8cyclictest30-21ksoftirqd/209:10:252
1488770994923,8cyclictest30-21ksoftirqd/212:25:192
1488770994910,8cyclictest30-21ksoftirqd/209:20:242
148876399491,13cyclictest1493916-21systemd-run08:50:011
148877499488,26cyclictest0-21swapper/310:45:023
1488774994832,12cyclictest1-21systemd07:55:023
1488774994828,16cyclictest275-21dbus-daemon11:50:013
148877099489,8cyclictest30-21ksoftirqd/210:14:522
148877099488,7cyclictest0-21swapper/212:36:262
1488770994840,5cyclictest1494984-21kworker/2:010:36:082
148877099481,6cyclictest0-21swapper/212:24:302
148876099482,5cyclictest1491797-21sshd08:07:250
1488774994725,16cyclictest275-21dbus-daemon11:55:013
1488774994725,16cyclictest275-21dbus-daemon11:55:003
1488774994711,5cyclictest3467718-21lldpd08:35:133
148877099478,8cyclictest0-21swapper/209:16:032
148877099474,8cyclictest30-21ksoftirqd/210:32:292
1488770994733,11cyclictest1491306-21systemctl08:00:002
1488770994712,9cyclictest0-21swapper/210:25:142
1488774994633,9cyclictest1-21systemd07:30:013
1488774994624,16cyclictest1-21systemd08:35:013
1488774994624,16cyclictest1-21systemd08:35:003
148877499461,3cyclictest131rcu_preempt10:17:373
148877099466,9cyclictest0-21swapper/212:01:432
148877099466,9cyclictest0-21swapper/212:01:422
148877099466,10cyclictest30-21ksoftirqd/212:32:252
148877099463,6cyclictest30-21ksoftirqd/209:46:272
1488770994634,8cyclictest1508214-21munin-plugin-st09:40:012
1488770994612,7cyclictest30-21ksoftirqd/209:41:472
148876099462,16cyclictest1488811-21wc07:10:020
148876099462,16cyclictest1488811-21wc07:10:010
1488774994528,11cyclictest0-21swapper/309:25:013
1488774994525,15cyclictest275-21dbus-daemon12:05:013
1488774994525,15cyclictest275-21dbus-daemon12:05:013
148877099451,8cyclictest1493120-21munin-run08:35:012
148877099451,8cyclictest1493120-21munin-run08:35:012
1488770994513,6cyclictest30-21ksoftirqd/210:52:302
148877099451,14cyclictest1565048-21ssh11:46:272
148876099451,6cyclictest364-21snmpd08:31:580
148876099451,6cyclictest364-21snmpd08:31:570
1488774994430,10cyclictest37-21ksoftirqd/309:10:023
1488774994428,10cyclictest0-21swapper/312:20:013
1488774994428,10cyclictest0-21swapper/312:20:013
1488774994416,8cyclictest37-21ksoftirqd/309:50:363
148877499441,4cyclictest131rcu_preempt10:11:183
1488774994410,20cyclictest1491290-21modprobe07:59:593
1488770994432,7cyclictest309-21in:imuxsock07:35:012
1488774994331,9cyclictest120-21kswapd012:15:003
1488774994331,9cyclictest120-21kswapd012:15:003
1488774994330,8cyclictest1546370-21rm11:04:593
1488774994329,12cyclictest37-21ksoftirqd/310:05:023
1488774994326,11cyclictest1490524-21sshd07:44:253
148877499432,3cyclictest131rcu_preempt11:14:063
148877499432,3cyclictest131rcu_preempt10:38:123
148877499431,3cyclictest131rcu_preempt09:40:023
148877099436,6cyclictest30-21ksoftirqd/209:00:012
148877099436,6cyclictest30-21ksoftirqd/209:00:002
1488770994334,4cyclictest1481936-21kworker/2:007:50:032
1488770994328,12cyclictest30-21ksoftirqd/209:55:512
1488774994230,8cyclictest1541836-21latency_hist10:55:033
1488774994225,12cyclictest3467602-21systemd-journal08:16:263
1488774994225,12cyclictest3467602-21systemd-journal08:16:263
1488774994225,12cyclictest1544175-21diskmemload11:00:023
1488774994215,17cyclictest0-21swapper/309:00:013
1488774994215,17cyclictest0-21swapper/309:00:003
148877099425,8cyclictest30-21ksoftirqd/211:14:132
1488774994125,13cyclictest3467602-21systemd-journal09:10:023
1488774994125,12cyclictest1503911-21ssh09:30:023
1488774994124,12cyclictest37-21ksoftirqd/308:14:593
1488774994124,12cyclictest3467602-21systemd-journal08:53:253
148877499412,3cyclictest131rcu_preempt09:42:113
148877099413,7cyclictest30-21ksoftirqd/211:58:542
148877099411,11cyclictest30-21ksoftirqd/208:10:002
148877499405,4cyclictest131rcu_preempt09:30:343
148877499404,3cyclictest131rcu_preempt10:25:023
1488774994028,9cyclictest120-21kswapd010:10:013
1488774994027,9cyclictest1492832-21sshd08:25:253
148877499402,3cyclictest131rcu_preempt11:39:033
1488774994022,14cyclictest3467602-21systemd-journal07:10:263
1488770994030,7cyclictest30-21ksoftirqd/208:45:012
1488770994030,7cyclictest30-21ksoftirqd/208:45:002
1488770994024,12cyclictest1490555-21latency_hist07:45:012
1488774993926,10cyclictest1586021-21sh12:33:143
1488774993923,12cyclictest3467602-21systemd-journal12:40:013
148877499391,3cyclictest131rcu_preempt11:40:253
148877499385,22cyclictest1489222-21modprobe07:19:593
1488774993829,7cyclictest37-21ksoftirqd/311:09:543
1488774993829,7cyclictest37-21ksoftirqd/311:09:533
1488774993826,6cyclictest1573882-21sh12:06:053
1488774993824,10cyclictest3467602-21systemd-journal08:24:243
1488774993822,11cyclictest37-21ksoftirqd/309:45:223
1488774993822,11cyclictest37-21ksoftirqd/308:05:023
148877099389,5cyclictest131rcu_preempt07:38:242
148877499377,6cyclictest131rcu_preempt09:19:243
1488774993725,10cyclictest37-21ksoftirqd/310:48:553
1488774993724,10cyclictest37-21ksoftirqd/311:24:453
148877499371,5cyclictest131rcu_preempt10:30:263
148876399372,10cyclictest1488775-21latency_hist07:10:021
148876399372,10cyclictest1488775-21latency_hist07:10:011
148877499363,4cyclictest131rcu_preempt11:30:063
148877499362,3cyclictest131rcu_preempt08:10:023
1488774993618,13cyclictest3467602-21systemd-journal09:05:003
148877499361,31cyclictest48-21kcompactd007:30:193
1488770993624,9cyclictest30-21ksoftirqd/208:45:022
148877099361,14cyclictest30-21ksoftirqd/207:53:082
1488774993523,9cyclictest1490849-21wc07:50:013
148877099353,8cyclictest30-21ksoftirqd/208:04:142
1488770993526,5cyclictest1493374-21sendmail08:39:592
1488770993523,9cyclictest30-21ksoftirqd/207:25:032
1488770993523,9cyclictest30-21ksoftirqd/207:25:022
1488770993522,10cyclictest1488960-21munin-plugin-st07:14:592
1488770993423,8cyclictest1492326-21cron08:20:012
1488770993423,8cyclictest1492326-21cron08:20:002
1488770993324,7cyclictest30-21ksoftirqd/208:30:002
1488774993218,11cyclictest1489511-21sh07:25:013
1488774993218,11cyclictest1489511-21sh07:25:003
1488770993121,8cyclictest30-21ksoftirqd/207:26:162
1488770993120,8cyclictest30-21ksoftirqd/208:52:182
1488770993117,11cyclictest275-21dbus-daemon09:05:012
1488774992916,8cyclictest1488772-21latency_hist07:10:023
1488774992916,8cyclictest1488772-21latency_hist07:10:013
1488770992919,7cyclictest30-21ksoftirqd/207:10:022
1488770992919,7cyclictest30-21ksoftirqd/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional