You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-07 - 15:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Apr 07, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1552680991201,3cyclictest0-21swapper/007:50:020
155268099951,77cyclictest0-21swapper/010:15:020
1552680998726,12cyclictest1627046-21ssh11:47:560
1552680998425,13cyclictest1612663-21ssh11:14:440
1552680998324,13cyclictest1597503-21ssh10:39:410
1552680998322,12cyclictest1607202-21ssh11:01:530
1552680998028,15cyclictest1562783-21ssh09:19:030
1552680998025,7cyclictest1645388-21ssh12:30:130
1552680998015,61cyclictest12-21ksoftirqd/011:42:250
155268099801,38cyclictest1641030-21ssh12:20:050
1552680997919,16cyclictest1634275-21ssh12:04:540
1552680997824,14cyclictest1637410-21ssh12:11:580
1552680997627,7cyclictest1589413-21ssh10:20:240
1552680997524,9cyclictest1598109-21ssh10:40:330
1552680997524,15cyclictest1640668-21ssh12:19:490
1552680997521,12cyclictest1634883-21ssh12:05:590
1552680997521,12cyclictest1634883-21ssh12:05:590
1552680997425,14cyclictest1581841-21ssh10:03:020
1552680997423,15cyclictest1630333-21ssh11:55:210
155268099734,6cyclictest0-21swapper/010:55:030
1552680997327,14cyclictest1579255-21ssh09:57:000
1552680997324,14cyclictest1583929-21ssh10:07:540
1552680997225,10cyclictest1629717-21ssh11:54:150
1552680997223,12cyclictest1572388-21ssh09:40:590
1552680997218,12cyclictest1574728-21ssh09:46:320
1552680997124,11cyclictest1576431-21ssh09:50:200
1552680997124,11cyclictest1576431-21ssh09:50:190
1552680997124,11cyclictest1556010-21systemd-run08:15:010
1552680997122,11cyclictest275-21dbus-daemon07:30:010
1552680997115,51cyclictest16850irq/134-800000009:40:000
1552680997034,33cyclictest16950irq/224-800000012:26:520
1552680996924,5cyclictest0-21swapper/010:50:000
1552680996924,16cyclictest1565101-21ssh09:24:280
1552680996919,12cyclictest1560896-21ssh09:14:490
1552680996914,13cyclictest1616033-21ssh11:22:160
1552683996859,5cyclictest1552688-21cat07:10:011
155268099681,13cyclictest0-21swapper/011:31:080
1552680996726,37cyclictest275-21dbus-daemon08:10:010
1552680996722,11cyclictest1649329-21ssh12:39:570
1552680996617,14cyclictest1566339-21ssh09:27:020
1552680996617,14cyclictest1566339-21ssh09:27:020
1552680996532,29cyclictest1554177-21sendmail07:40:000
1552680996529,5cyclictest1569130-21sh09:33:450
155268099652,4cyclictest1587385-21sshd10:15:410
155268099651,27cyclictest1605612-21ssh10:58:220
155268099651,27cyclictest1605612-21ssh10:58:220
155268099635,54cyclictest161764750ssh11:25:520
155268099635,54cyclictest161764750ssh11:25:520
1552680996330,29cyclictest1614355-21sh11:18:370
1552680996330,29cyclictest1614355-21sh11:18:370
1552680996319,14cyclictest1610092-21ssh11:08:410
1552680996319,14cyclictest1610092-21ssh11:08:400
1552680996224,8cyclictest1621780-21ssh11:35:250
1552680996224,14cyclictest1592510-21ssh10:27:520
1552680996220,13cyclictest1594838-21ssh10:33:250
1552690996051,6cyclictest1543007-21kworker/2:209:47:372
1552690996041,9cyclictest1558909-21diskmemload09:40:022
155268099602,5cyclictest1558910-21munin-run09:10:010
1552690995939,15cyclictest1-21systemd07:30:012
1552690995848,5cyclictest1591259-21kworker/2:110:34:152
1552690995847,6cyclictest1591259-21kworker/2:110:43:362
1552690995836,15cyclictest309-21in:imuxsock08:00:002
1552680995832,23cyclictest16950irq/224-800000009:00:010
1552690995749,5cyclictest1586897-21kworker/2:011:12:232
1552690995647,6cyclictest1558662-21kworker/2:310:07:192
1552690995646,6cyclictest1591259-21kworker/2:111:42:332
1552690995645,6cyclictest1591259-21kworker/2:112:30:022
1552690995644,6cyclictest1543007-21kworker/2:209:52:372
1552690995644,6cyclictest1543007-21kworker/2:209:52:372
1552690995547,5cyclictest1614976-21kworker/2:212:08:062
1552690995547,5cyclictest1614976-21kworker/2:212:08:062
1552690995519,5cyclictest131rcu_preempt08:15:012
1552680995522,5cyclictest1553234-21latency_hist07:20:030
1552680995515,25cyclictest16950irq/224-800000008:20:000
1552690995445,5cyclictest1591259-21kworker/2:112:16:042
1552690995445,5cyclictest1586897-21kworker/2:011:16:532
1552690995445,5cyclictest1586897-21kworker/2:011:16:532
1552694995332,15cyclictest275-21dbus-daemon10:05:013
1552690995345,5cyclictest1614976-21kworker/2:211:35:002
1552690995345,5cyclictest1543007-21kworker/2:210:01:452
1552690995345,5cyclictest1543007-21kworker/2:209:19:022
1552690995344,6cyclictest1614976-21kworker/2:211:28:072
1552690995344,6cyclictest1614976-21kworker/2:211:28:072
1552690995344,5cyclictest1614976-21kworker/2:212:35:182
1552690995344,5cyclictest1614976-21kworker/2:212:05:002
1552690995344,5cyclictest1591259-21kworker/2:110:49:592
1552690995344,5cyclictest1543007-21kworker/2:209:24:392
1552690995344,5cyclictest1543007-21kworker/2:209:10:122
1552680995338,11cyclictest1-21systemd08:35:010
1552680995327,9cyclictest12-21ksoftirqd/008:49:230
155269499528,8cyclictest37-21ksoftirqd/311:46:423
1552690995245,4cyclictest1586897-21kworker/2:011:02:192
1552690995244,5cyclictest1614976-21kworker/2:211:39:062
1552690995244,5cyclictest1543007-21kworker/2:209:45:012
1552690995243,5cyclictest1614976-21kworker/2:211:59:032
1552690995243,5cyclictest1591259-21kworker/2:112:21:282
1552690995242,6cyclictest1586897-21kworker/2:011:05:182
1552690995242,6cyclictest1586897-21kworker/2:011:05:182
1552690995236,12cyclictest3467602-21systemd-journal10:21:412
1552680995237,11cyclictest3467602-21systemd-journal08:00:010
1552690995143,5cyclictest1591259-21kworker/2:112:10:242
1552690995143,5cyclictest1591259-21kworker/2:111:50:352
1552690995143,5cyclictest1591259-21kworker/2:110:51:362
1552690995143,5cyclictest1558662-21kworker/2:310:10:032
1552690995143,5cyclictest1543007-21kworker/2:208:45:092
1552690995143,4cyclictest1591259-21kworker/2:111:20:432
1552690995142,6cyclictest1591259-21kworker/2:110:56:322
1552690995142,6cyclictest1591259-21kworker/2:110:56:322
1552690995142,5cyclictest1591259-21kworker/2:111:45:082
155268399513,3cyclictest131rcu_preempt08:00:001
1552683995111,6cyclictest131rcu_preempt10:32:511
1552694995024,6cyclictest37-21ksoftirqd/311:36:383
155269499501,9cyclictest1589457-21ssh10:20:303
1552690995042,5cyclictest1586897-21kworker/2:010:28:392
1552690995042,5cyclictest1558662-21kworker/2:310:16:322
1552690995042,5cyclictest1543007-21kworker/2:209:57:542
1552690995042,5cyclictest1543007-21kworker/2:208:05:012
1552690995041,6cyclictest1543007-21kworker/2:209:30:582
1552690995041,5cyclictest1543007-21kworker/2:207:40:032
1552683995033,12cyclictest3467602-21systemd-journal11:15:001
1552683995030,10cyclictest23-21ksoftirqd/111:37:011
155269499497,6cyclictest131rcu_preempt09:50:353
155269499497,6cyclictest131rcu_preempt09:50:353
1552694994933,12cyclictest275-21dbus-daemon12:20:013
1552690994941,5cyclictest1591259-21kworker/2:112:33:102
1552690994941,5cyclictest1591259-21kworker/2:110:37:192
1552690994941,5cyclictest1543007-21kworker/2:209:25:032
1552690994941,5cyclictest1543007-21kworker/2:209:25:022
1552690994941,5cyclictest1543007-21kworker/2:208:25:032
1552690994941,5cyclictest1543007-21kworker/2:208:25:022
1552690994940,6cyclictest1543007-21kworker/2:209:05:012
1552690994940,6cyclictest1543007-21kworker/2:207:17:462
1552694994838,6cyclictest1625785-21sh11:45:023
1552694994835,10cyclictest3467602-21systemd-journal07:27:373
1552694994832,12cyclictest3467602-21systemd-journal09:10:003
155268399481,3cyclictest131rcu_preempt10:40:011
155269499472,3cyclictest131rcu_preempt10:43:363
1552694994711,6cyclictest131rcu_preempt09:15:003
1552690994736,6cyclictest1543007-21kworker/2:207:50:002
155268099473,40cyclictest1558092-21sshd08:52:390
155269499463,3cyclictest131rcu_preempt09:20:013
1552694994611,6cyclictest0-21swapper/311:08:003
1552694994611,6cyclictest0-21swapper/311:08:003
155269499461,14cyclictest1642766-21ssh12:24:413
1552690994638,5cyclictest1543007-21kworker/2:208:09:422
155268399464,4cyclictest131rcu_preempt10:45:001
155268399464,3cyclictest131rcu_preempt12:20:001
155268399464,3cyclictest131rcu_preempt09:30:001
155268399464,3cyclictest131rcu_preempt09:30:001
155268399462,4cyclictest131rcu_preempt09:50:381
155268399462,4cyclictest131rcu_preempt09:50:381
1552683994621,10cyclictest0-21swapper/110:55:411
1552683994621,10cyclictest0-21swapper/110:55:411
155268399461,4cyclictest131rcu_preempt12:10:011
155268399461,4cyclictest131rcu_preempt12:10:011
1552694994532,9cyclictest1-21systemd09:50:003
1552690994535,6cyclictest1543007-21kworker/2:207:15:012
155268399454,4cyclictest131rcu_preempt11:22:041
155268399454,3cyclictest131rcu_preempt10:15:001
1552683994527,14cyclictest23-21ksoftirqd/109:25:021
155268399451,3cyclictest131rcu_preempt07:25:011
155269499447,8cyclictest37-21ksoftirqd/310:15:023
155269499446,7cyclictest37-21ksoftirqd/309:36:063
155269499444,3cyclictest131rcu_preempt10:15:033
155269499443,5cyclictest1604070-21ssh10:54:553
1552694994424,16cyclictest275-21dbus-daemon11:25:003
1552694994410,5cyclictest131rcu_preempt11:59:483
1552690994436,5cyclictest1543007-21kworker/2:208:20:022
1552690994436,5cyclictest1543007-21kworker/2:207:33:402
1552690994428,11cyclictest30-21ksoftirqd/209:00:022
155268399449,6cyclictest131rcu_preempt11:44:041
155268399449,5cyclictest131rcu_preempt11:05:221
155268399449,5cyclictest131rcu_preempt11:05:221
155268399444,3cyclictest131rcu_preempt07:20:001
1552683994431,10cyclictest1559036-21cut09:10:021
155268399442,3cyclictest131rcu_preempt10:50:331
155268399441,4cyclictest131rcu_preempt12:24:101
155268399441,4cyclictest131rcu_preempt09:31:501
155268399441,3cyclictest131rcu_preempt11:00:561
155268399441,3cyclictest131rcu_preempt09:44:381
155269499433,3cyclictest131rcu_preempt12:32:223
155269499433,3cyclictest131rcu_preempt10:45:043
1552694994328,12cyclictest37-21ksoftirqd/312:39:383
1552694994326,13cyclictest1595717-21grep10:35:023
155269499431,3cyclictest131rcu_preempt10:39:573
155268399437,5cyclictest131rcu_preempt11:47:021
155268399434,4cyclictest131rcu_preempt11:16:411
155268399434,4cyclictest131rcu_preempt11:16:401
155268399433,4cyclictest131rcu_preempt11:57:141
155268399433,3cyclictest131rcu_preempt12:01:031
155268399432,3cyclictest131rcu_preempt12:10:551
155268399432,3cyclictest131rcu_preempt10:26:081
155268399431,4cyclictest131rcu_preempt10:09:511
155268399431,4cyclictest131rcu_preempt09:55:531
155268399431,3cyclictest131rcu_preempt11:30:001
155268399431,3cyclictest131rcu_preempt11:29:591
155268399431,3cyclictest131rcu_preempt10:50:021
155268399431,3cyclictest131rcu_preempt10:00:491
1552680994336,4cyclictest16950irq/224-800000008:45:010
1552680994328,11cyclictest3467602-21systemd-journal08:22:380
155269499428,6cyclictest131rcu_preempt11:30:363
155269499428,6cyclictest131rcu_preempt10:07:313
1552694994231,8cyclictest1592080-21sshd10:26:423
1552694994230,9cyclictest37-21ksoftirqd/309:45:033
1552694994220,5cyclictest37-21ksoftirqd/310:56:033
1552694994220,5cyclictest37-21ksoftirqd/310:56:023
1552690994234,5cyclictest1543007-21kworker/2:207:50:042
1552690994233,5cyclictest1543007-21kworker/2:208:20:332
1552690994227,12cyclictest30-21ksoftirqd/209:05:002
1552690994227,11cyclictest30-21ksoftirqd/208:34:382
155268399426,6cyclictest131rcu_preempt11:53:531
1552683994227,11cyclictest3467602-21systemd-journal07:55:011
1552683994227,10cyclictest3467602-21systemd-journal10:20:001
155268399422,3cyclictest131rcu_preempt12:40:001
1552683994223,14cyclictest1558385-21systemctl09:00:011
155268399421,3cyclictest131rcu_preempt09:16:431
155268399421,3cyclictest131rcu_preempt09:12:551
155268399421,3cyclictest131rcu_preempt07:30:001
155268399421,16cyclictest1553660-21kworker/u8:108:20:021
155268099421,13cyclictest1557475-21cat08:40:020
155269499417,5cyclictest131rcu_preempt11:16:473
155269499417,5cyclictest131rcu_preempt11:16:473
155269499416,5cyclictest131rcu_preempt08:14:303
155269499413,6cyclictest131rcu_preempt12:25:283
1552694994128,10cyclictest37-21ksoftirqd/311:54:163
1552694994127,10cyclictest309-21in:imuxsock11:00:033
1552694994126,11cyclictest309-21in:imuxsock08:55:013
155269499411,36cyclictest1553211-21date07:20:023
1552690994128,7cyclictest1553422-21munin-plugin-st07:25:002
1552690994127,9cyclictest309-21in:imuxsock08:41:382
155268399414,3cyclictest131rcu_preempt08:40:001
155268399413,3cyclictest131rcu_preempt11:30:031
1552683994128,8cyclictest3467602-21systemd-journal07:40:011
1552683994127,9cyclictest23-21ksoftirqd/108:55:001
1552683994127,11cyclictest1554884-21wc07:50:021
155268399411,4cyclictest131rcu_preempt10:20:311
155268399411,3cyclictest131rcu_preempt09:46:001
1552680994131,7cyclictest12-21ksoftirqd/007:15:010
1552680994129,8cyclictest3467602-21systemd-journal07:20:040
155269499404,3cyclictest131rcu_preempt11:29:593
155269499404,3cyclictest131rcu_preempt11:29:593
155269499404,3cyclictest131rcu_preempt09:30:023
155269499404,3cyclictest131rcu_preempt09:30:023
1552694994026,9cyclictest0-21swapper/312:05:023
155269099405,25cyclictest1557334-21modprobe08:39:592
1552690994033,4cyclictest1543007-21kworker/2:208:54:012
1552683994030,7cyclictest23-21ksoftirqd/108:35:001
1552683994028,8cyclictest3467602-21systemd-journal07:15:011
1552683994025,11cyclictest1556679-21wc08:25:011
155268399402,3cyclictest131rcu_preempt12:31:221
155268399401,4cyclictest131rcu_preempt12:27:331
155268399401,4cyclictest131rcu_preempt09:05:001
155268399401,3cyclictest131rcu_preempt09:36:011
1552680994011,23cyclictest1552685-21latency_hist07:10:010
155269499393,3cyclictest131rcu_preempt09:56:153
1552694993931,5cyclictest1553924-21munin-plugin-st07:35:003
1552694993924,9cyclictest1558380-21run-parts09:00:003
155268399391,3cyclictest131rcu_preempt08:15:031
155268099391,35cyclictest0-21swapper/008:30:010
155268099391,35cyclictest0-21swapper/008:30:000
1552694993825,9cyclictest1-21systemd08:40:013
155269499381,3cyclictest131rcu_preempt09:20:333
155269499381,33cyclictest37-21ksoftirqd/309:31:343
155268099381,3cyclictest131rcu_preempt07:45:010
155268099381,33cyclictest0-21swapper/009:05:000
155269499376,4cyclictest131rcu_preempt11:11:433
155269499376,4cyclictest131rcu_preempt07:40:013
1552694993730,4cyclictest1-21systemd07:45:013
1552694993724,9cyclictest37-21ksoftirqd/307:55:013
1552694993724,10cyclictest37-21ksoftirqd/308:35:013
1552694993723,11cyclictest275-21dbus-daemon12:10:013
1552694993723,11cyclictest275-21dbus-daemon12:10:013
155269499371,7cyclictest0-21swapper/312:10:283
1552683993721,12cyclictest23-21ksoftirqd/107:44:591
155268399371,4cyclictest131rcu_preempt07:35:011
155269499367,4cyclictest131rcu_preempt07:57:383
1552694993618,14cyclictest3467602-21systemd-journal08:05:003
1552690993629,5cyclictest1543007-21kworker/2:207:41:502
1552683993624,8cyclictest1557628-21systemd-run08:45:011
1552680993622,10cyclictest3467718-21lldpd08:00:570
1552694993526,6cyclictest1556579-21latency_hist08:25:013
1552683993527,5cyclictest1556821-21cron08:30:011
1552683993527,5cyclictest1556821-21cron08:30:011
1552683993523,8cyclictest1555868-21date08:10:021
155268399351,3cyclictest131rcu_preempt08:49:591
1552680993528,4cyclictest1554113-21latency_hist07:35:020
155268099351,30cyclictest1554937-21sshd07:53:390
1552694993423,7cyclictest1-21systemd07:25:013
1552683993422,9cyclictest1555451-21date08:00:021
1552694993317,12cyclictest3467602-21systemd-journal08:50:003
155269499331,3cyclictest131rcu_preempt07:10:033
155269499322,5cyclictest346771850lldpd08:40:283
1552694993225,4cyclictest3467562-21sshd09:03:393
1552694993224,4cyclictest1556325-21sendmail08:20:003
1552694993223,6cyclictest1554710-21modprobe07:50:003
1552694993220,10cyclictest37-21ksoftirqd/308:30:033
1552694993220,10cyclictest37-21ksoftirqd/308:30:033
1552694993021,7cyclictest37-21ksoftirqd/307:13:493
1552694992917,9cyclictest37-21ksoftirqd/308:10:013
1552690992920,6cyclictest1552793-21cat07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional