You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 06:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 06, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
674283991441,7cyclictest0-21swapper/021:20:020
67428399143136,5cyclictest12-21ksoftirqd/020:25:030
674283991084,14cyclictest131rcu_preempt22:55:020
674283991071,63cyclictest1-21systemd23:55:020
674283991021,80cyclictest747421-21ssh23:32:010
67428399991,55cyclictest708829-21latency_hist22:10:030
674283999519,41cyclictest751438-21ssh23:40:370
674283999519,41cyclictest751438-21ssh23:40:370
674283999224,30cyclictest735272-21ssh23:06:080
674283999115,56cyclictest760933-21ssh00:00:530
674288998758,26cyclictest23-21ksoftirqd/123:10:011
674283998571,9cyclictest674286-21latency_hist19:10:020
67428399852,47cyclictest675734-21cut19:35:030
67428399841,50cyclictest0-21swapper/022:00:520
67428399836,45cyclictest737237-21ssh23:10:180
67428399823,46cyclictest692998-21ssh21:36:200
67428399823,40cyclictest763385-21modprobe00:06:170
674283998230,48cyclictest699550-21ssh21:50:090
67428399822,47cyclictest776762-21munin-run00:35:010
67428399822,47cyclictest776762-21munin-run00:35:000
67428399801,75cyclictest703585-21ssh21:59:090
67428399801,75cyclictest48-21kcompactd022:45:110
67428399801,55cyclictest150-21pr/ttyS200:14:570
674283997930,45cyclictest691692-21ssh21:33:530
674283997829,45cyclictest739475-21ssh23:15:050
67428399781,38cyclictest1-21systemd23:00:010
67428399772,38cyclictest699381-21cat21:50:010
674283997722,51cyclictest694928-21ssh21:40:210
674283997716,46cyclictest16850irq/134-800000000:39:060
674283997519,52cyclictest16850irq/134-800000022:30:230
674283997430,28cyclictest743305-21ssh23:23:290
674283997429,28cyclictest772362-21ssh00:25:120
67428399742,42cyclictest713475-21munin-run22:20:020
67428399742,42cyclictest713475-21munin-run22:20:010
67428399741,41cyclictest0-21swapper/000:21:130
67428399731,43cyclictest681183-21ssh21:11:120
674283997314,30cyclictest0-21swapper/023:56:120
674283997220,17cyclictest16850irq/134-800000021:18:260
674283997119,33cyclictest709029-21ssh22:10:200
67428399711,66cyclictest678867-21sshd20:37:260
67428399711,33cyclictest749268-21ssh23:35:580
674283997020,41cyclictest16850irq/134-800000022:23:530
67428399691,64cyclictest720112-21kworker/u8:122:42:500
67428399682,61cyclictest0-21swapper/022:40:020
67428399681,35cyclictest755214-21ssh23:49:100
674293996540,8cyclictest676572-21kworker/2:222:41:412
674283996523,26cyclictest768370-21ssh00:16:470
67428399651,59cyclictest718280-21cat22:30:020
67428399651,59cyclictest676525-21sshd19:50:170
674283996449,11cyclictest16950irq/224-800000020:00:000
674283996425,23cyclictest744598-21ssh23:25:550
67428399632,56cyclictest733170-21ssh23:01:390
67428399632,55cyclictest679421-21munin-run20:50:010
674283996319,10cyclictest131rcu_preempt21:10:000
67428399631,40cyclictest679646-21sshd20:52:170
67428399621,57cyclictest677565-21sshd20:11:170
67428399612,55cyclictest677062-21sshd20:01:180
674283996111,26cyclictest0-21swapper/021:25:400
674293996052,5cyclictest676572-21kworker/2:223:57:182
67428399602,54cyclictest675023-21munin-run19:25:000
67428399602,53cyclictest3467602-21systemd-journal19:25:190
67428399601,54cyclictest679929-21latency_hist21:00:000
67428399601,54cyclictest679929-21latency_hist21:00:000
67428399601,39cyclictest0-21swapper/019:20:020
67428399601,39cyclictest0-21swapper/019:20:020
67428399593,51cyclictest678638-21munin-run20:35:010
67428399593,51cyclictest678638-21munin-run20:35:010
67428399592,52cyclictest677355-21munin-run20:10:010
67428899582,52cyclictest687603-21ssh21:25:021
674293995721,6cyclictest680458-21diskmemload00:27:562
674293995637,13cyclictest1-21systemd23:35:002
67428399562,49cyclictest680182-21sshd21:02:310
67428399561,51cyclictest675760-21sshd19:39:170
674297995541,9cyclictest1-21systemd00:00:023
674293995547,5cyclictest676572-21kworker/2:221:51:332
67428399551,50cyclictest679144-21sshd20:42:170
67428399551,50cyclictest677810-21sshd20:15:180
67428399551,48cyclictest48-21kcompactd019:47:140
674297995418,10cyclictest0-21swapper/323:10:193
67428899545,38cyclictest744095-21ssh23:25:021
67429799532,4cyclictest131rcu_preempt22:46:193
67429799531,4cyclictest131rcu_preempt00:25:263
67429799531,3cyclictest131rcu_preempt22:43:503
674293995317,7cyclictest275-21dbus-daemon00:05:012
67428399531,48cyclictest364-21snmpd19:43:260
67428399531,48cyclictest364-21snmpd19:43:250
67429799524,4cyclictest131rcu_preempt22:21:153
67429799523,4cyclictest131rcu_preempt22:08:413
67429799522,4cyclictest131rcu_preempt23:38:503
67429799522,3cyclictest131rcu_preempt00:38:433
67429799521,3cyclictest131rcu_preempt22:04:533
674293995244,5cyclictest676572-21kworker/2:223:22:062
674293995244,5cyclictest676572-21kworker/2:222:45:482
674293995243,5cyclictest713474-21kworker/2:100:15:012
674293995238,9cyclictest754455-21sshd23:47:172
674288995230,16cyclictest3467602-21systemd-journal21:50:011
67428399522,46cyclictest678114-21sshd20:23:170
67429799516,3cyclictest131rcu_preempt00:30:033
67429799516,3cyclictest131rcu_preempt00:30:033
67429799515,4cyclictest131rcu_preempt00:06:393
67429799515,3cyclictest131rcu_preempt23:30:003
67429799514,3cyclictest131rcu_preempt23:22:223
67429799513,4cyclictest131rcu_preempt23:17:393
67429799513,3cyclictest131rcu_preempt21:48:293
674297995132,15cyclictest37-21ksoftirqd/323:45:013
674297995132,15cyclictest37-21ksoftirqd/323:45:013
67429799512,4cyclictest131rcu_preempt23:31:053
67429799512,4cyclictest131rcu_preempt22:34:453
67429799512,3cyclictest131rcu_preempt23:48:193
67429799511,4cyclictest131rcu_preempt00:21:073
674293995144,3cyclictest676572-21kworker/2:222:03:552
674293995143,5cyclictest713474-21kworker/2:100:15:362
674293995142,5cyclictest676572-21kworker/2:223:17:532
674293995123,18cyclictest30-21ksoftirqd/222:21:582
674293995113,9cyclictest131rcu_preempt21:18:132
674288995133,14cyclictest275-21dbus-daemon00:40:001
67428399511,46cyclictest364-21snmpd19:13:590
67429799503,4cyclictest131rcu_preempt23:00:473
67429799503,3cyclictest131rcu_preempt23:50:383
67429799503,3cyclictest131rcu_preempt21:59:403
67429799502,4cyclictest131rcu_preempt21:41:453
67429799502,3cyclictest131rcu_preempt00:10:183
67429799501,3cyclictest131rcu_preempt22:10:063
674297995011,10cyclictest0-21swapper/322:55:173
674293995042,5cyclictest713474-21kworker/2:100:07:582
674293995042,5cyclictest676572-21kworker/2:223:44:162
674293995042,5cyclictest676572-21kworker/2:223:44:162
674293995042,5cyclictest676572-21kworker/2:222:10:412
674293995042,5cyclictest676572-21kworker/2:200:34:262
674293995042,5cyclictest676572-21kworker/2:200:34:252
674293995030,14cyclictest275-21dbus-daemon21:50:002
67428899504,6cyclictest131rcu_preempt23:13:171
674288995040,7cyclictest674277-21cyclictest19:10:011
67429799495,3cyclictest131rcu_preempt22:55:023
67429799494,4cyclictest131rcu_preempt21:11:333
67429799492,3cyclictest131rcu_preempt23:09:123
67429799492,3cyclictest131rcu_preempt21:15:263
67429799491,4cyclictest131rcu_preempt22:37:533
674293994941,5cyclictest675549-21kworker/2:021:31:252
674293994940,5cyclictest676572-21kworker/2:222:25:152
674293994940,5cyclictest675549-21kworker/2:021:10:312
67429399491,9cyclictest778869-21ssh00:39:192
674297994835,9cyclictest1-21systemd19:30:003
67429799483,4cyclictest131rcu_preempt22:17:163
67429799483,4cyclictest131rcu_preempt22:17:163
67429799483,4cyclictest131rcu_preempt21:21:043
67429799483,4cyclictest131rcu_preempt00:01:123
67429799483,3cyclictest131rcu_preempt21:35:323
67429799482,3cyclictest131rcu_preempt22:28:293
67429799482,3cyclictest131rcu_preempt21:50:303
67429799482,3cyclictest131rcu_preempt21:27:553
67429799481,3cyclictest131rcu_preempt21:30:243
674293994840,5cyclictest676572-21kworker/2:221:21:122
674293994839,6cyclictest676572-21kworker/2:223:35:592
674293994833,7cyclictest676572-21kworker/2:222:30:352
674293994833,12cyclictest1-21systemd20:40:002
674293994824,7cyclictest676572-21kworker/2:223:53:572
674288994834,10cyclictest694515-21cron21:40:001
67428899483,3cyclictest131rcu_preempt20:15:001
67429799474,3cyclictest131rcu_preempt21:05:023
67429799473,4cyclictest131rcu_preempt00:18:303
67429799473,3cyclictest131rcu_preempt20:55:173
67429799473,3cyclictest131rcu_preempt20:55:173
674297994722,4cyclictest37-21ksoftirqd/321:10:013
674293994739,5cyclictest676572-21kworker/2:223:27:172
674293994739,5cyclictest676572-21kworker/2:222:07:132
674293994739,5cyclictest664145-21kworker/2:119:45:132
674293994712,7cyclictest131rcu_preempt22:19:002
674293994712,7cyclictest131rcu_preempt22:19:002
674288994735,8cyclictest1-21systemd21:00:021
674288994735,8cyclictest1-21systemd21:00:011
674288994734,10cyclictest750659-21sshd23:39:171
674288994732,10cyclictest3467602-21systemd-journal19:15:011
674288994730,13cyclictest677866-21run-parts20:20:001
674297994610,31cyclictest1-21systemd20:10:013
67429399469,27cyclictest30-21ksoftirqd/222:40:022
67429399466,7cyclictest30-21ksoftirqd/221:27:282
674293994639,4cyclictest676572-21kworker/2:222:56:112
674293994638,5cyclictest713474-21kworker/2:100:20:132
674293994638,5cyclictest676572-21kworker/2:221:58:282
674293994638,5cyclictest675549-21kworker/2:021:37:292
674293994630,12cyclictest3467602-21systemd-journal21:05:002
67429399462,13cyclictest694794-21ssh21:40:042
674288994631,11cyclictest275-21dbus-daemon21:20:011
674288994630,13cyclictest23-21ksoftirqd/100:00:291
674297994532,9cyclictest275-21dbus-daemon20:55:003
674297994529,11cyclictest37-21ksoftirqd/320:00:013
67429799452,3cyclictest131rcu_preempt20:43:183
674293994537,5cyclictest664145-21kworker/2:119:24:172
67428899453,4cyclictest131rcu_preempt23:16:021
674288994526,14cyclictest3467602-21systemd-journal20:47:171
674288994512,8cyclictest131rcu_preempt21:30:001
674293994437,4cyclictest675549-21kworker/2:019:55:012
674293994436,5cyclictest676572-21kworker/2:222:50:212
67428899444,3cyclictest131rcu_preempt00:20:021
674288994430,10cyclictest3467602-21systemd-journal23:00:001
674288994429,10cyclictest275-21dbus-daemon00:00:011
674288994428,12cyclictest23-21ksoftirqd/122:32:101
674297994328,11cyclictest275-21dbus-daemon20:35:023
674297994328,11cyclictest275-21dbus-daemon20:35:013
674293994335,5cyclictest676572-21kworker/2:223:03:522
674288994332,9cyclictest23-21ksoftirqd/122:04:491
674288994328,11cyclictest275-21dbus-daemon23:55:011
674288994327,12cyclictest23-21ksoftirqd/122:24:271
67429799424,3cyclictest150-21pr/ttyS219:38:173
67429799421,3cyclictest131rcu_preempt20:50:003
674293994233,5cyclictest664145-21kworker/2:119:18:172
674293994233,5cyclictest664145-21kworker/2:119:18:172
674293994229,9cyclictest1-21systemd20:30:012
674293994225,12cyclictest275-21dbus-daemon20:45:002
67429399422,3cyclictest131rcu_preempt23:05:322
674288994231,9cyclictest23-21ksoftirqd/121:43:131
674288994229,10cyclictest23-21ksoftirqd/122:15:401
674288994229,10cyclictest23-21ksoftirqd/122:15:401
674288994228,10cyclictest23-21ksoftirqd/122:52:231
674288994227,12cyclictest23-21ksoftirqd/122:44:051
674288994227,11cyclictest1-21systemd20:05:001
674288994227,10cyclictest3467602-21systemd-journal19:40:001
674288994212,20cyclictest0-21swapper/123:50:001
674288994210,5cyclictest131rcu_preempt19:20:171
67429799414,3cyclictest131rcu_preempt19:12:173
674297994132,7cyclictest37-21ksoftirqd/320:05:013
674297994125,12cyclictest3467602-21systemd-journal19:30:023
674293994133,5cyclictest675549-21kworker/2:020:20:022
67428899414,4cyclictest131rcu_preempt21:35:031
67428899413,3cyclictest131rcu_preempt20:30:011
674288994129,9cyclictest23-21ksoftirqd/120:08:171
674288994128,9cyclictest23-21ksoftirqd/100:34:531
674288994128,9cyclictest23-21ksoftirqd/100:34:521
674288994126,11cyclictest23-21ksoftirqd/119:30:011
67428899412,4cyclictest131rcu_preempt21:52:481
67429799404,3cyclictest131rcu_preempt19:20:023
67429799404,3cyclictest131rcu_preempt19:20:023
674297994024,12cyclictest275-21dbus-daemon19:50:003
674297994021,14cyclictest275-21dbus-daemon19:25:003
674297994020,15cyclictest275-21dbus-daemon20:25:013
67429399408,5cyclictest131rcu_preempt20:51:512
67429399405,4cyclictest131rcu_preempt23:10:492
674293994032,5cyclictest664145-21kworker/2:119:15:012
674288994029,9cyclictest23-21ksoftirqd/123:31:261
674288994028,10cyclictest23-21ksoftirqd/123:42:051
674288994028,10cyclictest23-21ksoftirqd/123:42:041
674297993919,5cyclictest37-21ksoftirqd/319:52:213
67429399392,28cyclictest291rcuc/220:10:022
67428899396,25cyclictest0-21swapper/123:25:361
67428899393,4cyclictest131rcu_preempt00:25:001
674288993929,8cyclictest23-21ksoftirqd/122:29:191
674288993928,7cyclictest679158-21cron20:45:001
674288993926,9cyclictest727277-21sshd22:49:171
674288993925,11cyclictest23-21ksoftirqd/122:11:021
674288993925,11cyclictest23-21ksoftirqd/121:10:361
674288993925,10cyclictest23-21ksoftirqd/100:10:081
674288993925,10cyclictest1-21systemd19:50:001
67428899392,4cyclictest131rcu_preempt00:27:411
67428899392,3cyclictest131rcu_preempt22:40:011
674288993921,11cyclictest309-21in:imuxsock20:35:001
674288993921,11cyclictest309-21in:imuxsock20:35:001
67428899391,5cyclictest131rcu_preempt19:35:021
67429799386,5cyclictest131rcu_preempt19:10:013
67429399384,3cyclictest131rcu_preempt20:50:022
674293993823,11cyclictest3467602-21systemd-journal20:05:002
674288993826,10cyclictest23-21ksoftirqd/121:55:131
674288993822,12cyclictest3467602-21systemd-journal20:40:001
674288993821,12cyclictest1-21systemd20:25:021
67428899381,3cyclictest131rcu_preempt23:02:551
67429399374,3cyclictest131rcu_preempt20:35:012
67429399374,3cyclictest131rcu_preempt20:35:012
67429399374,3cyclictest131rcu_preempt19:35:022
674293993729,5cyclictest678136-21munin-plugin-st20:25:002
674293993723,11cyclictest676914-21cat20:00:022
674293993723,10cyclictest275-21dbus-daemon20:15:012
674293993722,11cyclictest676025-21sshd19:41:172
674293993722,11cyclictest676025-21sshd19:41:172
67428899374,28cyclictest676814-21sh20:00:001
674288993729,6cyclictest23-21ksoftirqd/122:07:301
674288993728,7cyclictest23-21ksoftirqd/100:06:251
674288993724,10cyclictest23-21ksoftirqd/119:40:171
674288993724,10cyclictest23-21ksoftirqd/119:40:171
674288993722,11cyclictest23-21ksoftirqd/119:20:011
674288993722,11cyclictest23-21ksoftirqd/119:20:011
67428899371,5cyclictest131rcu_preempt20:55:011
674297993622,10cyclictest3467602-21systemd-journal19:44:173
674297993622,10cyclictest3467602-21systemd-journal19:44:173
67429399362,3cyclictest131rcu_preempt19:30:012
674297993524,9cyclictest37-21ksoftirqd/320:15:013
674297993423,8cyclictest37-21ksoftirqd/320:30:013
674293993426,5cyclictest664145-21kworker/2:119:39:212
67429399342,3cyclictest131rcu_preempt21:00:022
67429399342,3cyclictest131rcu_preempt21:00:012
67429399341,3cyclictest131rcu_preempt21:10:002
674288993426,6cyclictest23-21ksoftirqd/121:02:171
674297993320,9cyclictest679101-21sort20:40:023
67429799331,4cyclictest131rcu_preempt20:20:003
674288993327,4cyclictest676558-21latency_hist19:55:001
674288993317,12cyclictest275-21dbus-daemon21:10:001
67429399322,3cyclictest131rcu_preempt19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional