You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 09:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri Jun 26, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1589765999131,55cyclictest1642376-21ssh22:47:290
158976599811,62cyclictest1609562-21ssh21:38:290
1589765998030,34cyclictest1689249-21ssh00:26:110
1589765998020,55cyclictest16850irq/134-800000021:59:400
1589765997931,32cyclictest1621332-21ssh22:03:060
1589765997827,47cyclictest1658232-21ssh23:20:340
1589765997729,44cyclictest1599927-21ssh21:18:180
1589765997718,54cyclictest16850irq/134-800000021:42:450
1589771997621,21cyclictest1679788-21ssh00:06:181
1589765997631,29cyclictest1653327-21ssh23:10:210
1589765997631,29cyclictest1653327-21ssh23:10:210
158976599752,56cyclictest1692605-21ssh00:33:230
158976599743,34cyclictest1629232-21modprobe22:20:000
158976599742,67cyclictest1669949-21ssh23:45:220
158976599742,40cyclictest1676609-21ssh00:00:000
158976599741,57cyclictest1615440-21ssh21:50:310
158976599731,40cyclictest1598129-21ssh21:14:430
158976599731,40cyclictest1598129-21ssh21:14:430
158976599722,66cyclictest1633115-21ssh22:27:580
158976599722,66cyclictest0-21swapper/000:25:030
1589765997220,35cyclictest1662184-21ssh23:29:270
1589765997219,44cyclictest0-21swapper/023:05:240
1589771997126,13cyclictest1595940-21diskmemload00:33:361
158976599714,33cyclictest0-21swapper/019:15:010
1589765997137,30cyclictest1595397-21cron21:00:000
1589765997130,37cyclictest1646940-21ssh22:57:030
1589765997121,41cyclictest16850irq/134-800000000:01:220
1589765997119,43cyclictest16850irq/134-800000021:31:420
1589765997020,46cyclictest16850irq/134-800000023:35:420
1589765997020,45cyclictest16850irq/134-800000000:05:400
158976599701,65cyclictest1634327-21ssh22:30:140
158976599701,2cyclictest0-21swapper/022:39:110
1589776996933,31cyclictest30-21ksoftirqd/219:45:022
1589776996933,31cyclictest30-21ksoftirqd/219:45:022
158976599692,37cyclictest1674423-21date23:55:020
158976599692,34cyclictest1595159-21munin-run20:55:000
1589765996920,44cyclictest16850irq/134-800000023:40:200
158976599691,64cyclictest1591364-21latency_hist19:40:020
158976599691,63cyclictest1601864-21ssh21:22:140
1589771996820,14cyclictest1682331-21ssh00:11:361
158976599682,62cyclictest0-21swapper/000:38:320
158976599682,28cyclictest1686179-21munin-run00:20:000
1589765996820,39cyclictest16850irq/134-800000022:10:590
158976599681,52cyclictest1663624-21ssh23:32:170
1589765996723,40cyclictest1644727-21ssh22:52:250
1589765996719,43cyclictest16850irq/134-800000000:10:330
1589771996620,16cyclictest364-21snmpd23:21:501
1589765996620,41cyclictest16850irq/134-800000022:44:110
1589765996519,42cyclictest16850irq/134-800000021:49:300
1589765996519,42cyclictest16850irq/134-800000021:49:300
158976599651,2cyclictest0-21swapper/021:28:070
158976599643,52cyclictest163072650ssh22:22:540
1589765996433,27cyclictest1-21systemd20:05:010
158977699634,53cyclictest1598297-21unin-run21:15:012
158977699634,53cyclictest1598297-21unin-run21:15:012
1589776996325,34cyclictest30-21ksoftirqd/221:16:462
1589771996319,15cyclictest1599084-21ssh21:16:211
158977699624,51cyclictest1676601-40ogrotate23:59:592
1589771996226,24cyclictest1595940-21diskmemload22:21:581
158976599622,32cyclictest1657831-21ssh23:20:010
158976599621,41cyclictest1648848-21ssh23:00:510
1589776996126,31cyclictest275-21dbus-daemon00:25:012
1589765996134,24cyclictest1-21systemd20:20:010
158976599611,2cyclictest0-21swapper/021:10:000
158977699602,53cyclictest1663465-21ssh23:31:592
158977699602,52cyclictest1681396-21ssh00:09:582
158976599603,51cyclictest1592350-21ls20:00:020
158976599601,3cyclictest150-21pr/ttyS219:40:250
158976599601,3cyclictest150-21pr/ttyS219:40:250
158978099596,8cyclictest37-21ksoftirqd/300:40:013
158977699594,44cyclictest3467602-21systemd-journal22:35:012
1589776995934,22cyclictest1622148-21latency_hist22:05:022
158977699592,51cyclictest1594111-21cron20:35:002
1589776995917,38cyclictest1595114-21sed20:50:032
1589771995931,22cyclictest1653825-21ssh23:11:241
1589771995931,22cyclictest1653825-21ssh23:11:241
1589771995930,24cyclictest1616287-21ssh21:52:311
1589765995911,42cyclictest275-21dbus-daemon21:05:010
1589765995911,42cyclictest275-21dbus-daemon21:05:010
158977699583,51cyclictest1693327-21ssh00:35:002
158977699582,50cyclictest1683939-21latency_hist00:15:032
158977699581,54cyclictest1626752-21ssh22:14:412
1589776995815,39cyclictest1-21systemd23:40:002
1589771995836,17cyclictest1604320-21ssh21:27:251
1589771995831,23cyclictest1609969-21ssh21:39:291
158978099573,8cyclictest37-21ksoftirqd/323:50:003
158977699576,45cyclictest1631670-21modprobe22:25:002
158977699575,46cyclictest1629251-21sh22:20:002
158977699572,50cyclictest1668748-21ssh23:43:092
158977699572,50cyclictest1611051-21ssh21:41:232
158977699572,50cyclictest1594551-21cut20:40:022
158977699572,49cyclictest275-21dbus-daemon20:05:012
158977699572,16cyclictest1685381-21ssh00:18:072
158977699571,51cyclictest1616393-21ssh21:52:442
1589771995737,16cyclictest1671541-21ssh23:49:081
1589771995735,18cyclictest1679016-21ssh00:05:001
1589771995735,18cyclictest1657636-21ssh23:19:521
1589771995733,19cyclictest1660508-21ssh23:25:321
1589771995718,4cyclictest131rcu_preempt21:49:321
1589771995718,4cyclictest131rcu_preempt21:49:321
158976599573,26cyclictest1590771-21munin-run19:30:010
158976599572,50cyclictest0-21swapper/020:35:020
158976599572,49cyclictest1593721-21latency_hist20:25:020
158976599572,49cyclictest1593721-21latency_hist20:25:020
1589765995718,32cyclictest16850irq/134-800000022:07:100
158977699566,42cyclictest1619843-21ssh22:00:022
1589776995647,6cyclictest1579801-21kworker/2:123:22:522
158977699564,44cyclictest1615197-21latency_hist21:50:022
158977699564,41cyclictest1672019-21latency_hist23:50:022
158977699563,45cyclictest275-21dbus-daemon22:30:002
158977699562,49cyclictest1592264-21sshd19:56:122
158977699562,48cyclictest1637499-21sshd22:37:112
158977699562,48cyclictest1592842-21munin-run20:10:012
1589776995621,31cyclictest1660889-1kworker/u9:200:02:182
1589776995618,11cyclictest1694780-1kworker/u9:200:38:372
1589776995617,34cyclictest1690175-21sh00:28:172
1589771995635,16cyclictest1674487-21ssh23:55:051
1589771995632,18cyclictest1667584-21ssh23:40:231
1589771995630,22cyclictest1611399-21sh21:42:121
1589771995630,21cyclictest1617612-21ssh21:55:051
1589771995630,20cyclictest1689635-21ssh00:26:581
158976599561,27cyclictest1591008-21latency_hist19:35:020
158978099557,7cyclictest37-21ksoftirqd/321:16:533
158978099556,9cyclictest37-21ksoftirqd/322:15:253
158977699552,49cyclictest1624541-21true22:10:012
158977699552,47cyclictest1654665-21ssh23:13:292
158977699552,47cyclictest1654665-21ssh23:13:292
1589776995518,34cyclictest30-21ksoftirqd/219:25:012
1589776995518,31cyclictest30-21ksoftirqd/222:45:092
158977699551,51cyclictest1645932-21cut22:55:022
158977699551,49cyclictest1652953-21ssh23:09:532
158977699551,49cyclictest1605770-21ssh21:30:192
158977699551,49cyclictest1604786-21ssh21:28:332
158977699551,48cyclictest364-21snmpd22:42:012
1589771995540,10cyclictest275-21dbus-daemon20:50:011
1589771995540,10cyclictest275-21dbus-daemon20:50:011
1589771995538,13cyclictest1593748-21latency_hist20:25:021
1589771995538,13cyclictest1593748-21latency_hist20:25:021
1589771995536,15cyclictest1606922-21ssh21:33:031
1589771995533,18cyclictest1662825-21ssh23:30:261
1589771995533,18cyclictest1650487-21ssh23:04:411
1589771995532,19cyclictest1687753-21ssh00:23:101
1589771995532,18cyclictest1647350-21ssh22:57:561
1589771995531,20cyclictest1639813-21ssh22:42:011
1589771995531,18cyclictest1625500-21ssh22:11:551
158976599552,47cyclictest1594920-21/usr/sbin/munin20:50:010
158976599552,47cyclictest1594920-21/usr/sbin/munin20:50:010
158978099549,10cyclictest37-21ksoftirqd/321:56:533
1589780995446,5cyclictest361rcuc/321:45:173
1589780995446,5cyclictest361rcuc/321:45:173
1589780995420,21cyclictest1691063-21rm00:30:023
158977699544,45cyclictest1648199-21cron23:00:002
1589776995437,13cyclictest3467602-21systemd-journal23:05:012
158977699543,44cyclictest1590440-21sed19:20:022
158977699543,44cyclictest1590440-21sed19:20:022
158977699542,47cyclictest0-21swapper/221:21:192
1589776995412,37cyclictest275-21dbus-daemon21:00:012
158977699541,14cyclictest1657324-21sh23:19:032
1589771995435,14cyclictest1627687-21ssh22:16:301
1589771995433,17cyclictest1694092-21ssh00:36:191
1589771995433,17cyclictest1637587-21ssh22:37:221
1589771995433,16cyclictest1643953-21ssh22:50:371
1589771995431,18cyclictest1666152-21ssh23:37:391
1589771995430,19cyclictest1672256-21ssh23:50:231
1589771995430,19cyclictest1651348-21sh23:06:121
1589771995429,20cyclictest1623791-21ssh22:08:171
158976599541,49cyclictest1593099-21latency_hist20:15:020
158976599541,49cyclictest1-21systemd20:25:120
158978099536,7cyclictest37-21ksoftirqd/322:14:003
158977699533,45cyclictest1593857-21cron20:30:002
158977699532,47cyclictest1590988-21sshd19:32:542
1589776995320,29cyclictest1595151-21kworker/u8:021:37:582
158977699531,47cyclictest1673196-21ssh23:52:282
1589771995334,16cyclictest1642699-21sh22:48:111
1589771995334,15cyclictest1635061-21ssh22:31:571
1589771995333,16cyclictest1684587-21ssh00:16:181
1589771995331,18cyclictest1594845-21latency_hist20:45:021
1589771995330,18cyclictest1621531-21rm22:03:301
1589771995328,21cyclictest1595165-21systemd-run20:55:001
158976599532,30cyclictest1593032-21latency_hist20:10:020
158978099527,9cyclictest37-21ksoftirqd/322:21:363
158978099525,8cyclictest37-21ksoftirqd/322:25:073
158978099525,8cyclictest37-21ksoftirqd/321:44:243
158977699522,46cyclictest364-21snmpd19:30:002
158977699522,46cyclictest1614365-21ssh21:48:342
158977699522,46cyclictest0-21swapper/223:27:402
158977699522,45cyclictest1595662-21sshd21:02:122
158977699522,45cyclictest1595662-21sshd21:02:122
158977699522,45cyclictest1591250-21cron19:40:002
158977699521,44cyclictest1593720-21latency_hist20:25:022
158977699521,44cyclictest1593720-21latency_hist20:25:022
1589771995238,10cyclictest1-21systemd21:05:011
1589771995238,10cyclictest1-21systemd21:05:011
1589771995236,13cyclictest1632156-21ssh22:25:351
1589771995235,13cyclictest1601760-21ssh21:21:541
1589771995233,15cyclictest364-21snmpd20:35:011
1589771995231,16cyclictest1593168-21cat20:15:021
158976599528,39cyclictest275-21dbus-daemon19:55:010
158976599524,35cyclictest1591793-21modprobe19:50:000
158976599521,47cyclictest1590479-21sshd19:22:130
158978099516,8cyclictest37-21ksoftirqd/321:39:593
158978099516,7cyclictest37-21ksoftirqd/322:59:223
158978099516,11cyclictest37-21ksoftirqd/321:24:353
158978099514,7cyclictest37-21ksoftirqd/322:01:413
158978099514,32cyclictest1595940-21diskmemload00:00:033
158978099514,12cyclictest37-21ksoftirqd/322:45:013
158978099512,13cyclictest37-21ksoftirqd/322:08:023
158978099512,10cyclictest37-21ksoftirqd/322:52:553
1589780995113,5cyclictest131rcu_preempt23:01:453
158978099511,11cyclictest37-21ksoftirqd/323:14:313
158978099511,11cyclictest37-21ksoftirqd/323:14:313
158977699515,42cyclictest1-21systemd20:15:012
158977699515,40cyclictest1589858-21sort19:10:022
158977699515,40cyclictest1589858-21sort19:10:022
158977699515,38cyclictest1591794-21munin-plugin-st19:50:002
158977699513,36cyclictest1595166-21true20:55:002
158977699512,44cyclictest1595926-21cron21:10:002
1589771995123,23cyclictest1596279-21ssh21:10:131
1589771995123,23cyclictest1596279-21ssh21:10:131
158976599513,41cyclictest1590213-21cron19:20:000
158976599513,41cyclictest1590213-21cron19:20:000
158978099509,11cyclictest37-21ksoftirqd/322:33:243
158978099507,6cyclictest37-21ksoftirqd/323:20:113
158978099506,16cyclictest37-21ksoftirqd/321:27:213
158978099505,9cyclictest37-21ksoftirqd/319:25:003
1589780995038,9cyclictest37-21ksoftirqd/321:14:163
1589780995038,9cyclictest37-21ksoftirqd/321:14:163
1589780995037,11cyclictest37-21ksoftirqd/323:06:223
1589780995014,9cyclictest37-21ksoftirqd/323:19:543
158977699503,42cyclictest0-21swapper/219:10:092
1589771995034,12cyclictest1590771-21unin-run19:30:011
1589771995034,11cyclictest1590507-21munin-run19:25:001
1589771995031,14cyclictest1591142-21date19:35:031
1589771995030,16cyclictest1591954-21sed19:50:021
158976599501,16cyclictest1594641-21sshd20:44:120
158978099495,8cyclictest37-21ksoftirqd/323:28:593
158978099495,14cyclictest37-21ksoftirqd/320:40:013
158977699492,43cyclictest364-21snmpd20:45:162
158977699491,44cyclictest1593574-21sed20:20:022
1589771994933,12cyclictest1589943-21sshd19:12:511
1589771994931,13cyclictest1595944-21munin-run21:10:001
1589771994930,15cyclictest1593511-21cat20:20:021
158978099486,11cyclictest37-21ksoftirqd/300:23:193
158978099485,11cyclictest37-21ksoftirqd/323:40:283
1589771994833,12cyclictest1592092-21latency_hist19:55:011
1589771994832,10cyclictest275-21dbus-daemon20:00:011
1589771994831,14cyclictest1594343-21sshd20:36:121
1589771994829,15cyclictest1591565-21wc19:45:021
1589771994829,15cyclictest1591565-21wc19:45:021
1589771994827,16cyclictest1592792-21sshd20:05:121
158978099477,8cyclictest37-21ksoftirqd/321:52:363
158978099477,10cyclictest37-21ksoftirqd/300:18:433
158978099476,9cyclictest37-21ksoftirqd/323:50:343
158978099475,9cyclictest37-21ksoftirqd/322:36:493
158978099475,7cyclictest37-21ksoftirqd/322:46:233
158978099475,7cyclictest37-21ksoftirqd/320:45:013
1589780994736,9cyclictest37-21ksoftirqd/321:31:383
1589780994734,9cyclictest3467602-21systemd-journal20:30:003
158977699473,40cyclictest0-21swapper/219:55:002
158977699473,39cyclictest1594814-21latency_hist20:45:022
1589771994734,9cyclictest3467602-21systemd-journal20:03:271
1589771994732,11cyclictest364-21snmpd20:25:121
1589771994731,12cyclictest1595418-21munin-run21:00:001
1589771994729,13cyclictest1591270-21munin-run19:40:001
158978099465,9cyclictest37-21ksoftirqd/323:37:123
158978099465,8cyclictest37-21ksoftirqd/300:00:423
158976599464,38cyclictest1594362-21cron20:40:000
158976599461,42cyclictest1589761-21cyclictest19:10:020
158976599461,42cyclictest1589761-21cyclictest19:10:020
158978099455,6cyclictest37-21ksoftirqd/323:33:003
158978099454,10cyclictest37-21ksoftirqd/300:12:423
1589771994529,11cyclictest1589857-21ls19:10:021
1589771994529,11cyclictest1589857-21ls19:10:021
158978099447,8cyclictest37-21ksoftirqd/300:31:373
158978099446,10cyclictest37-21ksoftirqd/300:07:483
1589780994430,11cyclictest37-21ksoftirqd/319:50:003
1589771994320,18cyclictest1590279-21latency_hist19:20:021
1589771994320,18cyclictest1590279-21latency_hist19:20:011
1589780994129,8cyclictest0-21swapper/321:10:013
158978099412,8cyclictest37-21ksoftirqd/320:35:003
1589780994032,5cyclictest361rcuc/319:15:003
1589780994030,6cyclictest1591069-21latency_hist19:35:023
1589780993825,10cyclictest3467602-21systemd-journal20:10:123
1589780993825,10cyclictest1593607-21cron20:25:003
1589780993825,10cyclictest1593607-21cron20:25:003
1589780993822,11cyclictest275-21dbus-daemon21:00:013
1589780993729,5cyclictest1590229-21sendmail19:20:003
1589780993729,5cyclictest1590229-21sendmail19:20:003
1589780993725,8cyclictest1591506-21sshd19:40:123
1589780993725,8cyclictest1591506-21sshd19:40:123
1589780993724,10cyclictest37-21ksoftirqd/320:20:003
158978099371,9cyclictest0-21swapper/321:01:353
158978099371,9cyclictest0-21swapper/321:01:343
1589780993629,5cyclictest37-21ksoftirqd/320:00:023
1589780993629,4cyclictest274-21cron20:55:003
1589780993526,6cyclictest309-21in:imuxsock20:05:003
1589780993525,5cyclictest309-21in:imuxsock19:30:013
1589780993425,6cyclictest37-21ksoftirqd/319:40:023
1589780993425,6cyclictest1592060-21pluginstate19:55:013
1589780993423,8cyclictest37-21ksoftirqd/320:50:013
1589780993423,8cyclictest37-21ksoftirqd/320:50:013
1589780993023,4cyclictest1592948-21latency_hist20:10:023
1589780993020,7cyclictest1589812-21wc19:10:023
1589780993020,7cyclictest1589812-21wc19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional