You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 08:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Mar 25, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1619030999527,36cyclictest1624209-21latency_hist20:50:022
1619030999527,36cyclictest1624209-21latency_hist20:50:022
1619023999386,4cyclictest150-21pr/ttyS219:15:020
1619030998925,32cyclictest1658912-21sh22:28:272
161903099831,44cyclictest1662908-21ssh22:37:542
161903099812,4cyclictest131rcu_preempt00:00:032
161903099812,4cyclictest131rcu_preempt00:00:032
161903099792,45cyclictest1691065-21sh23:43:322
161903099792,45cyclictest1691065-21sh23:43:322
1619030997911,33cyclictest1693732-21latency_hist23:50:022
161903099762,43cyclictest1706842-21date00:20:032
161903099712,31cyclictest1652539-21ssh22:13:372
1619030997010,27cyclictest0-21swapper/200:23:012
161903099691,64cyclictest1677933-21ssh23:12:552
161903099691,42cyclictest364-21snmpd20:06:372
161903099691,38cyclictest1625530-21ssh21:10:092
161903099691,38cyclictest1625530-21ssh21:10:082
161903099682,61cyclictest1679277-21ssh23:15:422
161903099681,63cyclictest1688821-21ssh23:38:152
161903099681,63cyclictest1684466-21ssh23:28:022
161903099673,59cyclictest1709485-21sshd00:26:212
161903099673,59cyclictest1709485-21sshd00:26:212
161903099662,59cyclictest1622350-21munin-run20:15:022
161903099661,61cyclictest1698311-21ssh00:00:082
161903099661,61cyclictest1630684-21ssh21:22:292
161903099661,60cyclictest1703592-21ssh00:12:512
161903099652,58cyclictest1640946-21sshd21:46:202
1619030996524,37cyclictest1712023-21sh00:32:232
161903099651,60cyclictest1713735-21ssh00:36:162
161903099651,60cyclictest1686705-21ssh23:33:192
161903099651,60cyclictest1686705-21ssh23:33:192
161903099651,60cyclictest1676573-21ssh23:09:522
161903099651,60cyclictest1670056-21ssh22:54:432
161903099651,60cyclictest1665874-21ssh22:44:582
161903099651,60cyclictest1661012-21ssh22:33:312
161903099651,60cyclictest1649118-21ssh22:05:232
161903099651,60cyclictest1649118-21ssh22:05:232
161903099651,60cyclictest1628555-21ssh21:17:302
161903099642,58cyclictest1653394-21ssh22:15:182
161903099641,59cyclictest1701235-21ssh00:07:172
161903099641,59cyclictest1694107-21ssh23:50:232
161903099641,59cyclictest1681965-21ssh23:22:072
161903099641,59cyclictest1657092-21ssh22:24:202
161903099641,59cyclictest1647803-21ssh22:02:292
161903099641,59cyclictest1644912-21ssh21:55:262
161903099641,59cyclictest1643567-21ssh21:52:372
161903099641,59cyclictest1638588-21ssh21:40:422
161903099632,57cyclictest1666587-21ssh22:46:092
161903099631,58cyclictest1670501-21ssh22:55:142
161903099631,58cyclictest1636516-21ssh21:35:532
161903099631,58cyclictest1634702-21ssh21:31:492
161903099631,58cyclictest1632023-21ssh21:25:162
161903099631,58cyclictest1622551-21sshd20:15:212
161903099631,58cyclictest1620777-21sshd19:43:202
161903099631,58cyclictest1620777-21sshd19:43:202
161903099622,56cyclictest1672591-21ssh23:00:062
161903099621,57cyclictest1619991-21sshd19:27:202
1619023996234,25cyclictest16850irq/134-800000000:40:020
161903099611,56cyclictest1620247-21sshd19:33:202
161903099602,54cyclictest1624701-21sshd20:59:202
161903099592,52cyclictest364-21snmpd20:40:322
161903099591,54cyclictest1621025-21sshd19:47:192
161902699599,13cyclictest23-21ksoftirqd/122:58:541
161902399592,3cyclictest131rcu_preempt00:18:350
161903099582,51cyclictest364-21snmpd19:35:182
161903099582,51cyclictest1619525-21munin-run19:20:022
161903099581,53cyclictest48-21kcompactd020:25:462
161903099581,53cyclictest1624433-21sshd20:51:202
1619023995827,23cyclictest1619019-21cyclictest19:20:010
1619023995819,35cyclictest16850irq/134-800000021:10:020
161903099571,52cyclictest1622856-21sshd20:23:192
1619026995717,10cyclictest23-21ksoftirqd/123:20:011
1619023995719,33cyclictest16850irq/134-800000000:20:430
161903099561,51cyclictest1619775-21munin-run19:25:012
161902699569,5cyclictest131rcu_preempt22:54:561
1619026995610,11cyclictest23-21ksoftirqd/122:03:471
1619023995619,32cyclictest16850irq/134-800000022:53:270
161902399561,3cyclictest131rcu_preempt00:09:130
161903099552,48cyclictest48-21kcompactd020:36:182
161903099551,50cyclictest3467562-21sshd19:10:192
161902699556,14cyclictest23-21ksoftirqd/123:25:071
1619026995511,10cyclictest23-21ksoftirqd/122:24:511
1619023995528,12cyclictest1674464-21ssh23:04:590
1619023995525,15cyclictest1693498-21ssh23:49:200
1619023995520,31cyclictest16850irq/134-800000023:15:350
1619023995519,31cyclictest16850irq/134-800000021:48:530
161903099542,47cyclictest1621512-21date19:55:022
161903099542,47cyclictest1621512-21date19:55:012
1619030995423,27cyclictest30-21ksoftirqd/219:55:022
161903099541,49cyclictest1623429-21munin-run20:35:002
161903099541,48cyclictest364-21snmpd21:06:032
1619026995414,9cyclictest23-21ksoftirqd/100:23:011
1619026995410,10cyclictest23-21ksoftirqd/100:33:321
161902399544,46cyclictest0-21swapper/023:30:020
161902399544,46cyclictest0-21swapper/023:30:020
1619023995427,12cyclictest1639583-21ssh21:43:150
1619023995420,30cyclictest16850irq/134-800000023:22:140
1619023995420,30cyclictest16850irq/134-800000023:13:340
1619023995419,30cyclictest16850irq/134-800000022:04:260
1619023995418,31cyclictest16850irq/134-800000022:34:330
1619023995418,31cyclictest16850irq/134-800000021:26:580
1619023995415,15cyclictest131rcu_preempt23:56:550
1619023995415,15cyclictest131rcu_preempt23:56:550
161903099531,48cyclictest1624961-21sshd21:00:202
161902699536,7cyclictest23-21ksoftirqd/123:21:331
161902699536,13cyclictest23-21ksoftirqd/121:19:151
161902699534,8cyclictest0-21swapper/100:15:031
161902699533,15cyclictest23-21ksoftirqd/122:07:051
161902699533,15cyclictest23-21ksoftirqd/122:07:041
161902699533,13cyclictest23-21ksoftirqd/120:50:021
161902699533,13cyclictest23-21ksoftirqd/120:50:021
1619023995326,12cyclictest1712245-21ssh00:33:030
1619023995326,12cyclictest1709871-21ssh00:27:240
1619023995326,12cyclictest1709871-21ssh00:27:240
161902399532,16cyclictest275-21dbus-daemon22:45:010
1619023995319,30cyclictest16850irq/134-800000022:47:140
1619023995319,30cyclictest16850irq/134-800000021:17:380
1619023995318,31cyclictest16850irq/134-800000023:51:080
1619023995318,31cyclictest16850irq/134-800000022:38:560
1619023995318,31cyclictest16850irq/134-800000022:16:560
1619023995318,31cyclictest16850irq/134-800000021:50:310
1619023995318,31cyclictest16850irq/134-800000000:12:000
1619023995317,31cyclictest16850irq/134-800000022:10:360
161903599528,6cyclictest131rcu_preempt21:30:013
161902699528,11cyclictest23-21ksoftirqd/122:18:071
161902699527,9cyclictest23-21ksoftirqd/123:14:341
161902699526,8cyclictest23-21ksoftirqd/121:35:011
161902699525,7cyclictest23-21ksoftirqd/123:46:381
161902699525,7cyclictest23-21ksoftirqd/123:32:591
161902699525,7cyclictest23-21ksoftirqd/123:32:591
161902699524,9cyclictest23-21ksoftirqd/122:43:091
1619026995232,15cyclictest1638108-21cron21:40:011
1619023995224,13cyclictest1688483-21ssh23:37:270
1619023995219,30cyclictest16850irq/134-800000023:26:340
1619023995219,29cyclictest1629873-21sh21:20:200
1619023995218,30cyclictest16850irq/134-800000023:43:230
1619023995218,30cyclictest16850irq/134-800000023:43:230
1619023995218,30cyclictest16850irq/134-800000022:07:050
1619023995218,30cyclictest16850irq/134-800000022:07:050
1619023995217,31cyclictest16850irq/134-800000022:25:090
1619023995217,31cyclictest16850irq/134-800000021:37:060
1619023995217,31cyclictest16850irq/134-800000021:32:550
1619023995216,31cyclictest16850irq/134-800000021:56:110
161902399521,29cyclictest131rcu_preempt19:25:020
161903099511,46cyclictest364-21snmpd20:02:192
161902699515,10cyclictest23-21ksoftirqd/123:54:541
161902699514,4cyclictest131rcu_preempt00:00:001
161902699514,4cyclictest131rcu_preempt00:00:001
161902699514,13cyclictest23-21ksoftirqd/121:13:091
161902699514,13cyclictest23-21ksoftirqd/121:13:091
161902699513,11cyclictest23-21ksoftirqd/120:35:011
161902699512,12cyclictest23-21ksoftirqd/122:11:261
1619026995111,10cyclictest23-21ksoftirqd/122:40:021
1619023995117,31cyclictest16850irq/134-800000022:57:480
1619023995116,31cyclictest16850irq/134-800000021:13:440
1619023995116,31cyclictest16850irq/134-800000021:13:430
161903599505,4cyclictest131rcu_preempt20:20:003
161903599503,26cyclictest1625251-21diskmemload22:10:033
161903599503,26cyclictest1625251-21diskmemload22:10:033
161903099501,45cyclictest1619083-21cat19:10:022
161902699506,11cyclictest23-21ksoftirqd/121:51:211
161902699506,10cyclictest23-21ksoftirqd/121:45:571
161902699503,7cyclictest23-21ksoftirqd/122:27:411
161902699502,11cyclictest23-21ksoftirqd/121:40:541
1619026995012,5cyclictest131rcu_preempt21:21:351
161902399509,36cyclictest0-21swapper/021:00:010
1619023995023,12cyclictest1655771-21ssh22:20:520
1619023995017,29cyclictest16850irq/134-800000023:09:110
1619023995016,30cyclictest16850irq/134-800000000:00:350
161903599499,6cyclictest131rcu_preempt21:19:593
1619035994930,14cyclictest1-21systemd23:00:013
161902699498,6cyclictest131rcu_preempt23:09:481
161902699497,7cyclictest23-21ksoftirqd/122:47:061
161902699496,7cyclictest23-21ksoftirqd/121:26:571
161902699495,9cyclictest23-21ksoftirqd/100:02:191
161902699495,8cyclictest23-21ksoftirqd/121:58:321
161902699495,10cyclictest23-21ksoftirqd/123:00:111
1619023994925,18cyclictest3467602-21systemd-journal19:55:000
161903599483,5cyclictest131rcu_preempt23:52:073
1619035994832,13cyclictest3467602-21systemd-journal00:00:043
161902699489,11cyclictest23-21ksoftirqd/100:17:291
161902699487,11cyclictest23-21ksoftirqd/100:36:351
161902699486,7cyclictest23-21ksoftirqd/123:41:551
161902699486,7cyclictest23-21ksoftirqd/123:41:551
161902699486,7cyclictest23-21ksoftirqd/100:26:441
161902699486,7cyclictest23-21ksoftirqd/100:26:441
161902699481,6cyclictest23-21ksoftirqd/120:40:011
1619035994734,9cyclictest37-21ksoftirqd/321:40:013
1619035994728,14cyclictest275-21dbus-daemon20:15:013
161902699474,6cyclictest23-21ksoftirqd/122:31:341
161902699474,13cyclictest23-21ksoftirqd/120:25:031
161902699473,7cyclictest23-21ksoftirqd/123:37:001
1619026994735,8cyclictest1621539-21cron20:00:011
1619026994735,8cyclictest1621539-21cron20:00:011
1619026994728,13cyclictest275-21dbus-daemon19:45:011
1619026994728,13cyclictest275-21dbus-daemon19:45:001
161902699471,13cyclictest23-21ksoftirqd/121:10:001
161903599463,3cyclictest131rcu_preempt00:16:233
1619035994630,12cyclictest309-21in:imuxsock00:30:013
1619035994630,12cyclictest309-21in:imuxsock00:30:013
1619035994628,13cyclictest275-21dbus-daemon20:00:013
1619035994628,13cyclictest275-21dbus-daemon20:00:013
161903599462,6cyclictest131rcu_preempt23:50:013
1619035994615,6cyclictest346771850lldpd23:32:403
1619035994615,6cyclictest346771850lldpd23:32:403
1619026994631,12cyclictest23-21ksoftirqd/120:55:001
1619023994639,4cyclictest16950irq/224-800000020:00:010
1619023994639,4cyclictest16950irq/224-800000020:00:010
161902699457,11cyclictest23-21ksoftirqd/120:05:011
161902699456,10cyclictest23-21ksoftirqd/119:13:201
1619026994529,11cyclictest309-21in:imuxsock19:50:001
161902699452,7cyclictest23-21ksoftirqd/100:06:271
1619035994432,9cyclictest1661952-21sshd22:35:213
161903599442,5cyclictest131rcu_preempt22:32:413
1619035994419,5cyclictest37-21ksoftirqd/321:46:223
161902699443,9cyclictest23-21ksoftirqd/119:30:011
1619026994427,13cyclictest3467602-21systemd-journal20:30:011
1619023994437,5cyclictest12-21ksoftirqd/019:43:200
1619023994437,5cyclictest12-21ksoftirqd/019:43:190
161903599434,4cyclictest131rcu_preempt22:49:173
1619035994320,5cyclictest37-21ksoftirqd/300:24:043
1619035994319,6cyclictest37-21ksoftirqd/300:38:043
161902399436,31cyclictest1622904-21kworker/0:020:40:000
1619023994338,3cyclictest16950irq/224-800000020:35:000
1619023994329,9cyclictest3467602-21systemd-journal20:30:000
161902399431,37cyclictest0-21swapper/020:20:000
161902399431,34cyclictest16202401sshd19:31:220
161903599429,6cyclictest131rcu_preempt21:52:393
161903599424,5cyclictest131rcu_preempt23:24:073
161903599424,3cyclictest131rcu_preempt20:35:023
161903599424,33cyclictest150-21pr/ttyS220:26:403
161903599423,6cyclictest131rcu_preempt22:28:233
161903599422,3cyclictest131rcu_preempt23:03:353
1619035994218,7cyclictest37-21ksoftirqd/300:32:263
1619035994218,5cyclictest37-21ksoftirqd/323:10:473
1619035994216,7cyclictest37-21ksoftirqd/323:41:223
1619035994216,7cyclictest37-21ksoftirqd/323:41:223
1619035994216,6cyclictest37-21ksoftirqd/321:33:193
161902699421,7cyclictest0-21swapper/119:21:191
161902699421,10cyclictest23-21ksoftirqd/119:35:011
1619023994228,10cyclictest12-21ksoftirqd/020:05:000
1619023994227,11cyclictest1622349-21systemd-run20:15:000
1619023994226,12cyclictest1622275-21cat20:10:020
161902399421,6cyclictest151rcuc/020:50:020
161902399421,6cyclictest151rcuc/020:50:010
161902399421,11cyclictest1622875-21munin-plugin-st20:25:010
161903599415,6cyclictest0-21swapper/321:14:053
161903599415,6cyclictest0-21swapper/321:14:053
161903599415,3cyclictest131rcu_preempt22:15:213
161903599413,4cyclictest131rcu_preempt23:15:033
1619035994120,5cyclictest37-21ksoftirqd/323:39:093
1619035994120,5cyclictest37-21ksoftirqd/322:21:273
1619035994119,5cyclictest37-21ksoftirqd/323:59:433
1619035994119,5cyclictest37-21ksoftirqd/323:59:423
1619035994119,5cyclictest37-21ksoftirqd/321:42:463
1619035994116,6cyclictest37-21ksoftirqd/319:30:013
161902699416,10cyclictest23-21ksoftirqd/120:15:021
161902699412,9cyclictest0-21swapper/120:10:021
161902699411,6cyclictest0-21swapper/120:58:481
161903599406,5cyclictest131rcu_preempt00:12:043
161903599405,4cyclictest131rcu_preempt23:29:303
1619035994026,9cyclictest275-21dbus-daemon21:05:013
1619035994025,12cyclictest275-21dbus-daemon19:20:003
161903599401,6cyclictest131rcu_preempt21:10:013
161902699403,8cyclictest23-21ksoftirqd/120:45:001
161903599394,3cyclictest131rcu_preempt23:05:333
161903599394,31cyclictest150-21pr/ttyS219:36:563
161903599393,3cyclictest131rcu_preempt22:14:443
1619035993930,7cyclictest37-21ksoftirqd/321:57:163
1619035993927,10cyclictest37-21ksoftirqd/322:03:583
1619035993920,5cyclictest37-21ksoftirqd/322:51:013
1619035993918,16cyclictest275-21dbus-daemon21:00:013
1619035993915,6cyclictest37-21ksoftirqd/300:05:563
1619023993928,8cyclictest1621182-21date19:50:020
1619023993925,10cyclictest1619985-21sshd19:26:200
1619035993828,7cyclictest37-21ksoftirqd/319:55:023
1619035993817,4cyclictest37-21ksoftirqd/322:40:063
1619035993815,5cyclictest37-21ksoftirqd/321:21:323
161903599375,3cyclictest131rcu_preempt20:06:203
1619026993727,6cyclictest309-21in:imuxsock19:20:011
1619023993721,12cyclictest1623925-21sshd20:40:200
161902399371,33cyclictest0-21swapper/021:01:160
161903599364,3cyclictest131rcu_preempt19:10:203
161903599363,3cyclictest131rcu_preempt20:50:003
161903599363,3cyclictest131rcu_preempt20:50:003
161902699366,8cyclictest23-21ksoftirqd/119:51:341
161902699365,8cyclictest23-21ksoftirqd/121:04:201
1619035993522,10cyclictest37-21ksoftirqd/319:50:003
1619026993521,10cyclictest1-21systemd20:20:001
1619023993523,8cyclictest1620506-21cron19:40:000
1619023993521,10cyclictest364-21snmpd20:51:450
161903599344,4cyclictest131rcu_preempt19:40:213
161903599344,4cyclictest131rcu_preempt19:40:213
161903599344,3cyclictest131rcu_preempt19:33:193
1619035993420,10cyclictest37-21ksoftirqd/320:25:013
1619035993315,4cyclictest37-21ksoftirqd/320:00:513
161902699331,5cyclictest150-21pr/ttyS219:35:101
161903599323,3cyclictest131rcu_preempt19:21:003
1619035993222,6cyclictest37-21ksoftirqd/320:40:013
161902399327,15cyclictest0-21swapper/019:10:010
1619035993124,4cyclictest1618968-21latency_hist19:10:023
1619035993120,8cyclictest37-21ksoftirqd/320:41:263
161903599303,3cyclictest131rcu_preempt20:51:203
161902699272,20cyclictest1619025-21date19:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional