You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 17:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 21, 2026 12:45:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
156566499911,6cyclictest0-21swapper/009:35:020
156567499855,6cyclictest30-21ksoftirqd/211:20:022
156567499855,6cyclictest30-21ksoftirqd/211:20:022
1565664998520,12cyclictest1665726-21ssh12:25:290
156566499841,66cyclictest1644666-21ssh11:41:390
1565664998125,52cyclictest1620963-21ssh10:52:060
1565664998123,40cyclictest1595746-21ssh09:59:510
1565664998123,40cyclictest1595746-21ssh09:59:500
156566499792,60cyclictest1581328-21ssh09:29:400
1565670997629,43cyclictest3467602-21systemd-journal11:18:181
1565670997629,43cyclictest3467602-21systemd-journal11:18:181
1565664997628,31cyclictest1630159-21ssh11:11:220
1565664997627,32cyclictest1622921-21ssh10:56:130
1565664997625,47cyclictest1593067-21ssh09:54:070
1565664997623,36cyclictest1658001-21ssh12:09:460
1565664997623,36cyclictest1658001-21ssh12:09:450
1565664997535,37cyclictest16950irq/224-800000012:00:380
1565664997528,28cyclictest1636519-21ssh11:24:530
156566499751,70cyclictest1638669-21ssh11:29:210
156566499751,70cyclictest1638669-21ssh11:29:210
1565664997428,28cyclictest1651890-21ssh11:56:460
1565664997425,34cyclictest1576272-21ssh09:19:130
1565664997424,33cyclictest1669693-21ssh12:34:150
1565664997424,31cyclictest1659151-21ssh12:12:000
1565664997326,29cyclictest1619782-21ssh10:49:570
1565664997326,29cyclictest1619782-21ssh10:49:560
1565664997315,12cyclictest1597164-21ssh10:02:330
1565664997311,57cyclictest16850irq/134-800000010:32:490
1565664997311,11cyclictest1626534-21ssh11:03:560
1565670997229,39cyclictest275-21dbus-daemon11:30:021
1565670997229,39cyclictest275-21dbus-daemon11:30:011
1565664997226,42cyclictest0-21swapper/011:50:080
1565664997220,18cyclictest1601716-21ssh10:12:020
1565664997219,9cyclictest1605278-21ssh10:19:470
156567499705,5cyclictest131rcu_preempt10:10:032
156566499702,63cyclictest1641306-21ssh11:34:580
1565664997024,30cyclictest1662491-21ssh12:19:060
156566499701,52cyclictest0-21swapper/009:41:040
1565664996926,26cyclictest1589784-21ssh09:47:090
1565664996926,26cyclictest1589784-21ssh09:47:090
1565664996921,14cyclictest1627219-21ssh11:05:030
156566499691,64cyclictest0-21swapper/011:37:280
156566499691,64cyclictest0-21swapper/011:37:280
1565664996914,8cyclictest1578431-21ssh09:23:300
1565664996914,8cyclictest1578431-21ssh09:23:300
1565670996824,5cyclictest1641967-21sh11:36:011
1565670996824,5cyclictest1641967-21sh11:36:011
156566499682,15cyclictest1613215-21ssh10:35:560
156566499682,15cyclictest1613215-21ssh10:35:550
156567099671,6cyclictest131rcu_preempt10:57:311
1565664996722,41cyclictest1599188-21ssh10:06:340
1565670996628,12cyclictest3467602-21systemd-journal12:09:181
1565670996628,12cyclictest3467602-21systemd-journal12:09:181
1565670996625,4cyclictest1593178-21sh09:54:201
1565670996624,4cyclictest1599075-21sh10:06:201
156566499666,5cyclictest16850irq/134-800000009:37:510
156566499663,8cyclictest1571925-21latency_hist09:10:030
156566499662,59cyclictest0-21swapper/012:37:170
1565664996624,37cyclictest1634147-21modprobe11:20:000
1565664996624,37cyclictest1634147-21modprobe11:20:000
1565664996620,11cyclictest275-21dbus-daemon08:50:000
1565664996615,12cyclictest1608699-21ssh10:26:320
156567099643,14cyclictest1576680-21systemctl09:20:001
1565670996424,4cyclictest1618674-21sh10:47:211
1565670996424,4cyclictest1618674-21sh10:47:201
1565664996443,4cyclictest16850irq/134-800000012:25:000
1565670996326,6cyclictest23-21ksoftirqd/109:10:061
1565670996326,6cyclictest23-21ksoftirqd/109:10:061
156566499631,57cyclictest0-21swapper/010:20:590
156567099622,20cyclictest1569229-21modprobe08:20:001
1565670996215,12cyclictest275-21dbus-daemon11:15:011
1565664996225,32cyclictest1569301-21cat08:20:020
1565664996225,31cyclictest1567913-21sshd07:52:180
1565664996210,48cyclictest16850irq/134-800000011:48:230
1565664996210,46cyclictest16850irq/134-800000009:11:530
1565664996210,46cyclictest16850irq/134-800000009:11:530
1565670996125,4cyclictest1606321-21sh10:21:361
1565670996120,9cyclictest23-21ksoftirqd/109:44:551
156567899592,44cyclictest1566886-21kworker/u8:107:50:023
156567499597,9cyclictest30-21ksoftirqd/212:00:022
156567499591,5cyclictest30-21ksoftirqd/211:30:012
156567499591,5cyclictest30-21ksoftirqd/211:30:012
1565670995922,7cyclictest309-21in:imuxsock10:10:021
1565670995921,7cyclictest23-21ksoftirqd/110:02:241
1565670995921,10cyclictest23-21ksoftirqd/111:35:001
156566499591,53cyclictest0-21swapper/010:41:590
156567499589,11cyclictest30-21ksoftirqd/207:20:002
156567499585,9cyclictest30-21ksoftirqd/211:05:002
1565670995824,8cyclictest23-21ksoftirqd/111:06:421
1565670995819,9cyclictest1507997-1kworker/u9:109:22:321
1565670995819,9cyclictest1507997-1kworker/u9:109:22:311
156566499582,19cyclictest1570553-21munin-plugin-st08:45:000
1565664995814,30cyclictest1565665-21latency_hist07:10:020
1565664995814,30cyclictest1565665-21latency_hist07:10:020
1565674995714,5cyclictest131rcu_preempt10:47:102
1565674995714,5cyclictest131rcu_preempt10:47:102
1565670995724,8cyclictest23-21ksoftirqd/112:03:561
1565670995723,4cyclictest1583136-21sh09:33:251
1565664995721,2cyclictest0-21swapper/008:40:020
156567499562,11cyclictest30-21ksoftirqd/212:30:042
156567099562,13cyclictest3467602-21systemd-journal08:40:011
1565664995610,40cyclictest0-21swapper/007:50:000
1565674995546,7cyclictest30-21ksoftirqd/210:55:022
156567099553,18cyclictest1567677-21cron07:50:011
1565664995526,23cyclictest1-21systemd09:00:000
1565664995525,26cyclictest12-21ksoftirqd/007:45:010
156567499548,8cyclictest30-21ksoftirqd/212:38:552
1565674995446,5cyclictest1570816-21kworker/2:309:09:182
1565674995445,6cyclictest30-21ksoftirqd/209:53:412
156567499541,12cyclictest3467602-21systemd-journal09:28:182
1565670995429,22cyclictest1581224-21sh09:29:271
1565670995422,5cyclictest309-21in:imuxsock11:41:181
1565670995420,9cyclictest23-21ksoftirqd/111:57:301
1565678995337,13cyclictest37-21ksoftirqd/308:55:023
1565678995337,13cyclictest37-21ksoftirqd/308:55:023
1565674995318,12cyclictest30-21ksoftirqd/211:00:012
1565674995318,11cyclictest30-21ksoftirqd/209:15:022
1565674995318,11cyclictest30-21ksoftirqd/209:15:022
1565674995315,13cyclictest30-21ksoftirqd/208:00:012
156567099533,21cyclictest1571922-21cat09:10:021
156567099533,15cyclictest0-21swapper/112:11:051
1565670995324,4cyclictest1635513-21sh11:22:371
1565670995323,8cyclictest23-21ksoftirqd/112:37:471
1565664995324,25cyclictest1569838-21latency_hist08:30:020
1565664995318,8cyclictest16950irq/224-800000007:23:360
156567499528,9cyclictest30-21ksoftirqd/209:41:052
156567499526,10cyclictest30-21ksoftirqd/210:40:352
156567499525,11cyclictest30-21ksoftirqd/209:59:192
156567499525,11cyclictest30-21ksoftirqd/209:59:192
1565674995243,6cyclictest30-21ksoftirqd/210:20:022
156567499522,7cyclictest30-21ksoftirqd/209:50:022
156567499522,7cyclictest30-21ksoftirqd/209:50:012
1565674995224,8cyclictest30-21ksoftirqd/209:39:032
1565674995223,7cyclictest30-21ksoftirqd/209:32:202
1565664995224,25cyclictest1571022-21sshd08:50:180
1565664995224,25cyclictest1571022-21sshd08:50:180
1565678995133,14cyclictest275-21dbus-daemon09:20:003
156567899511,7cyclictest0-21swapper/310:15:023
156567499517,9cyclictest30-21ksoftirqd/211:53:292
156567499517,8cyclictest30-21ksoftirqd/210:25:022
156567499517,10cyclictest30-21ksoftirqd/212:03:232
156567499516,16cyclictest30-21ksoftirqd/208:10:002
156567499515,9cyclictest30-21ksoftirqd/212:11:232
156567499514,13cyclictest30-21ksoftirqd/212:25:022
1565674995110,9cyclictest30-21ksoftirqd/211:05:232
1565670995125,7cyclictest23-21ksoftirqd/107:39:241
1565670995123,8cyclictest23-21ksoftirqd/110:31:181
1565670995113,10cyclictest0-21swapper/111:51:111
1565664995121,8cyclictest16950irq/224-800000007:28:370
156567499506,11cyclictest30-21ksoftirqd/211:20:082
1565674995042,5cyclictest1567198-21kworker/2:208:50:162
1565674995042,5cyclictest1567198-21kworker/2:208:50:162
1565674995038,8cyclictest30-21ksoftirqd/212:08:292
1565674995038,8cyclictest30-21ksoftirqd/212:08:282
156567499503,6cyclictest30-21ksoftirqd/210:05:022
1565674995016,9cyclictest30-21ksoftirqd/210:15:012
1565664995028,19cyclictest1569648-21sed08:25:020
1565664995017,4cyclictest1565855-21sh07:15:000
1565678994938,6cyclictest1672407-21cron12:40:013
156567499497,12cyclictest30-21ksoftirqd/210:31:192
1565674994941,5cyclictest1631779-21kworker/2:112:19:152
1565674994940,6cyclictest1566870-21kworker/2:007:32:362
1565674994928,16cyclictest275-21dbus-daemon09:00:002
156567499491,8cyclictest1641336-21systemd-run11:35:012
1565674994913,11cyclictest30-21ksoftirqd/211:45:002
1565670994912,6cyclictest131rcu_preempt11:05:011
156567899481,8cyclictest131rcu_preempt12:20:103
156567499486,13cyclictest30-21ksoftirqd/211:35:342
156567499486,13cyclictest30-21ksoftirqd/211:35:342
1565674994841,4cyclictest1567198-21kworker/2:209:15:452
1565674994840,5cyclictest1570816-21kworker/2:311:14:342
1565674994840,5cyclictest1567198-21kworker/2:208:15:042
1565674994840,5cyclictest1566870-21kworker/2:008:26:102
1565674994840,5cyclictest1566870-21kworker/2:008:20:432
1565674994840,5cyclictest1555429-21kworker/2:307:36:592
1565674994840,5cyclictest1555429-21kworker/2:307:22:422
1565674994839,6cyclictest1566870-21kworker/2:008:30:412
1565674994838,4cyclictest1567198-21kworker/2:207:54:022
156567499482,12cyclictest30-21ksoftirqd/210:37:322
156567499482,12cyclictest30-21ksoftirqd/210:37:322
156567499482,12cyclictest30-21ksoftirqd/209:20:532
156567499482,12cyclictest30-21ksoftirqd/209:20:522
1565674994811,13cyclictest30-21ksoftirqd/212:30:032
156567099485,31cyclictest221rcuc/108:50:021
156567099483,9cyclictest3467602-21systemd-journal10:36:141
156567099483,9cyclictest3467602-21systemd-journal10:36:131
1565670994831,13cyclictest275-21dbus-daemon08:00:011
1565664994841,4cyclictest16950irq/224-800000008:05:010
1565664994841,4cyclictest16950irq/224-800000008:05:010
156566499482,14cyclictest1568206-21modprobe08:00:000
1565664994816,14cyclictest1-21systemd07:35:010
1565678994732,10cyclictest3467602-21systemd-journal11:05:003
1565678994728,13cyclictest0-21swapper/310:40:013
1565678994728,13cyclictest0-21swapper/310:40:003
1565678994720,21cyclictest37-21ksoftirqd/310:30:013
156567499475,7cyclictest30-21ksoftirqd/210:20:132
1565674994739,5cyclictest1631779-21kworker/2:111:47:332
1565674994739,5cyclictest1567198-21kworker/2:208:49:012
1565674994739,5cyclictest1567198-21kworker/2:208:40:052
1565674994739,5cyclictest1567198-21kworker/2:208:12:482
1565674994739,5cyclictest1566870-21kworker/2:008:35:332
1565674994739,5cyclictest1558860-21kworker/2:007:11:152
1565674994738,6cyclictest1567198-21kworker/2:208:01:292
1565674994738,6cyclictest1567198-21kworker/2:208:01:292
1565674994738,6cyclictest1555429-21kworker/2:307:25:242
156566499477,12cyclictest0-21swapper/007:40:030
1565678994632,11cyclictest37-21ksoftirqd/310:02:163
1565678994631,11cyclictest1-21systemd11:50:023
1565678994630,13cyclictest37-21ksoftirqd/311:35:013
1565678994630,12cyclictest1663012-21date12:20:023
1565674994639,5cyclictest1566870-21kworker/2:007:40:122
1565674994639,4cyclictest1570816-21kworker/2:309:03:262
1565674994639,4cyclictest1567198-21kworker/2:207:45:152
156567099465,3cyclictest150-21pr/ttyS210:19:091
156567099462,17cyclictest1568460-21cron08:05:001
156567099462,17cyclictest1568460-21cron08:05:001
1565678994530,11cyclictest1579105-21sh09:25:003
1565678994530,11cyclictest1579105-21sh09:25:003
1565678994527,14cyclictest275-21dbus-daemon12:00:013
1565670994531,10cyclictest1586411-21sed09:40:021
1565670994521,7cyclictest23-21ksoftirqd/112:28:341
1565670994512,5cyclictest131rcu_preempt10:45:001
1565664994530,11cyclictest12-21ksoftirqd/007:20:000
1565678994431,10cyclictest37-21ksoftirqd/311:08:023
156567099443,8cyclictest1589493-21ssh09:46:251
156567099443,8cyclictest1589493-21ssh09:46:241
156567099442,4cyclictest131rcu_preempt12:23:521
1565664994428,12cyclictest1-21systemd09:05:010
1565678994332,7cyclictest309-21in:imuxsock12:10:023
1565678994332,7cyclictest309-21in:imuxsock12:10:013
1565678994328,10cyclictest275-21dbus-daemon10:00:013
1565678994328,10cyclictest275-21dbus-daemon10:00:013
156567899431,38cyclictest37-21ksoftirqd/311:21:193
1565674994331,8cyclictest30-21ksoftirqd/207:10:022
1565674994331,8cyclictest30-21ksoftirqd/207:10:022
156567099431,6cyclictest0-21swapper/110:52:371
156567099431,12cyclictest1594539-21ssh09:57:031
156567099431,12cyclictest1594539-21ssh09:57:031
156566499431,23cyclictest0-21swapper/008:33:470
156567899424,6cyclictest131rcu_preempt08:00:003
1565678994232,7cyclictest37-21ksoftirqd/311:20:003
1565678994232,7cyclictest37-21ksoftirqd/311:20:003
1565678994229,8cyclictest37-21ksoftirqd/309:40:013
1565678994218,6cyclictest37-21ksoftirqd/310:44:003
156567899421,5cyclictest131rcu_preempt09:30:343
156567099423,9cyclictest1662333-21rm12:18:461
156567099421,11cyclictest364-21snmpd07:13:331
156566499422,4cyclictest131rcu_preempt08:10:000
1565678994130,8cyclictest1637753-21sshd11:27:193
1565678994130,8cyclictest1637753-21sshd11:27:183
1565678994129,9cyclictest37-21ksoftirqd/310:30:493
1565678994127,12cyclictest37-21ksoftirqd/310:59:233
1565678994127,11cyclictest37-21ksoftirqd/310:08:253
1565678994125,12cyclictest37-21ksoftirqd/309:44:553
156567099418,6cyclictest131rcu_preempt08:10:011
156567099417,4cyclictest131rcu_preempt07:30:001
156567099415,5cyclictest1646397-21ssh11:45:031
1565670994128,8cyclictest1567425-21cron07:45:001
1565670994128,10cyclictest23-21ksoftirqd/110:30:001
1565664994117,5cyclictest12-21ksoftirqd/008:15:010
156567899406,4cyclictest131rcu_preempt12:25:423
156567899404,5cyclictest131rcu_preempt09:45:203
156567899404,5cyclictest131rcu_preempt09:45:203
1565678994030,8cyclictest37-21ksoftirqd/307:30:013
1565678994027,8cyclictest1581480-21grep09:30:003
1565678994026,10cyclictest3466785-21systemd-udevd09:11:593
1565678994026,10cyclictest3466785-21systemd-udevd09:11:593
1565678994019,4cyclictest37-21ksoftirqd/311:54:303
1565678994018,6cyclictest131rcu_preempt08:45:023
156567099402,7cyclictest1668929-21ssh12:32:281
156567099402,4cyclictest131rcu_preempt08:15:011
1565678993929,6cyclictest1619463-21sh10:49:173
1565678993929,6cyclictest1619463-21sh10:49:163
1565678993928,8cyclictest37-21ksoftirqd/312:32:593
1565678993927,9cyclictest37-21ksoftirqd/307:25:003
1565678993926,10cyclictest37-21ksoftirqd/311:40:263
1565670993928,7cyclictest1566108-21cron07:20:001
156567899384,5cyclictest131rcu_preempt12:03:193
156567899383,5cyclictest131rcu_preempt10:17:103
1565678993828,7cyclictest1631782-21sh11:15:023
1565678993826,10cyclictest37-21ksoftirqd/309:50:213
1565678993825,10cyclictest37-21ksoftirqd/308:30:013
1565678993817,18cyclictest37-21ksoftirqd/310:52:263
156567099384,3cyclictest131rcu_preempt08:30:011
1565670993824,10cyclictest23-21ksoftirqd/108:25:001
156567899376,23cyclictest361rcuc/312:15:013
1565678993729,4cyclictest1569254-21sendmail08:20:013
156567899372,7cyclictest131rcu_preempt07:20:013
1565678993725,9cyclictest309-21in:imuxsock08:55:023
1565678993718,8cyclictest131rcu_preempt11:39:593
1565678993718,8cyclictest131rcu_preempt11:39:583
1565678993718,15cyclictest3467602-21systemd-journal07:55:003
1565678993717,15cyclictest311-21rs:main3
156567099374,27cyclictest1565673-21sort07:10:021
156567099374,27cyclictest1565673-21sort07:10:021
1565670993725,9cyclictest23-21ksoftirqd/109:04:401
156567099371,5cyclictest131rcu_preempt08:45:021
1565678993627,5cyclictest1571981-21sh09:10:023
1565678993617,8cyclictest131rcu_preempt10:23:433
1565678993531,2cyclictest0-21swapper/307:45:023
1565678993527,5cyclictest1567129-21sshd07:37:183
1565678993522,9cyclictest1571789-21sed09:05:023
1565678993522,5cyclictest309-21in:imuxsock08:10:013
1565670993525,7cyclictest275-21dbus-daemon08:35:011
1565678993424,7cyclictest37-21ksoftirqd/307:15:023
1565678993424,5cyclictest37-21ksoftirqd/308:47:183
1565678993422,9cyclictest1-21systemd08:25:003
1565678993325,5cyclictest1570194-21latency_hist08:35:023
1565678993319,11cyclictest37-21ksoftirqd/308:40:003
156567899331,8cyclictest150-21pr/ttyS208:05:023
156567899331,8cyclictest150-21pr/ttyS208:05:023
1565670993325,4cyclictest1571253-21latency_hist08:55:021
1565670993320,10cyclictest309-21in:imuxsock07:55:001
156567899324,21cyclictest1568979-21cut08:15:003
156567099321,4cyclictest131rcu_preempt08:55:011
1565670993116,10cyclictest0-21swapper/109:00:001
1565670993021,7cyclictest23-21ksoftirqd/107:24:181
1565670992922,4cyclictest1566975-21latency_hist07:35:021
156567899282,5cyclictest131rcu_preempt07:10:033
156567899282,5cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional