You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 11:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 27, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3299622998367,12cyclictest12-21ksoftirqd/023:00:020
3299622997633,5cyclictest16850irq/134-800000022:40:010
3299625997433,10cyclictest23-21ksoftirqd/100:39:251
3299629996813,9cyclictest30-21ksoftirqd/223:35:012
3299629996721,9cyclictest30-21ksoftirqd/221:50:022
3299625996727,9cyclictest23-21ksoftirqd/100:05:021
3299632996612,43cyclictest3344312-21rm22:40:023
3299622996657,3cyclictest150-21pr/ttyS219:15:010
329963299645,11cyclictest37-21ksoftirqd/320:19:593
3299622996455,7cyclictest16950irq/224-800000022:19:230
329963299636,12cyclictest37-21ksoftirqd/300:30:023
3299629996316,12cyclictest30-21ksoftirqd/200:04:132
3299625996116,9cyclictest23-21ksoftirqd/100:30:011
3299622996128,13cyclictest3380252-21ssh00:05:000
3299625996017,11cyclictest23-21ksoftirqd/122:25:011
3299625996017,10cyclictest23-21ksoftirqd/121:20:021
3299625996015,9cyclictest23-21ksoftirqd/123:05:011
3299629995912,8cyclictest30-21ksoftirqd/221:20:022
329962599599,5cyclictest131rcu_preempt21:25:011
329962599596,10cyclictest23-21ksoftirqd/122:20:011
3299622995926,16cyclictest3323627-21ssh21:51:120
329963299586,9cyclictest37-21ksoftirqd/322:00:013
3299629995810,9cyclictest30-21ksoftirqd/219:35:022
3299629995810,8cyclictest30-21ksoftirqd/222:15:012
329962599584,11cyclictest23-21ksoftirqd/122:09:031
329963299575,10cyclictest37-21ksoftirqd/322:25:063
3299629995715,6cyclictest30-21ksoftirqd/223:27:222
3299629995714,9cyclictest30-21ksoftirqd/223:00:532
329962599577,9cyclictest23-21ksoftirqd/122:05:011
329962599577,9cyclictest23-21ksoftirqd/121:43:381
329962599577,9cyclictest23-21ksoftirqd/121:43:371
329962599577,11cyclictest23-21ksoftirqd/122:47:431
329962599577,11cyclictest23-21ksoftirqd/121:48:371
329962599571,8cyclictest23-21ksoftirqd/123:00:001
3299622995725,4cyclictest131rcu_preempt20:35:000
3299622995716,36cyclictest3305856-21cut21:10:020
329963299567,6cyclictest131rcu_preempt22:25:013
329963299567,15cyclictest37-21ksoftirqd/321:40:003
329963299565,8cyclictest37-21ksoftirqd/300:37:593
329963299565,8cyclictest37-21ksoftirqd/300:00:033
3299632995614,6cyclictest131rcu_preempt19:45:033
3299629995637,13cyclictest1-21systemd19:40:012
3299629995627,11cyclictest30-21ksoftirqd/200:14:092
3299629995627,11cyclictest30-21ksoftirqd/200:14:092
3299629995618,9cyclictest131rcu_preempt22:22:522
3299629995612,8cyclictest30-21ksoftirqd/221:00:012
3299629995611,10cyclictest30-21ksoftirqd/221:40:002
3299629995610,10cyclictest30-21ksoftirqd/200:19:402
329962599567,7cyclictest23-21ksoftirqd/100:24:381
329962599567,7cyclictest23-21ksoftirqd/100:24:381
329962599566,14cyclictest23-21ksoftirqd/100:10:301
329962599566,14cyclictest23-21ksoftirqd/100:10:291
3299625995615,11cyclictest23-21ksoftirqd/120:15:021
329962299567,3cyclictest0-21swapper/022:05:030
3299622995644,7cyclictest16950irq/224-800000021:49:590
329962299564,13cyclictest1-21systemd00:10:010
329963299555,9cyclictest37-21ksoftirqd/323:30:013
329963299552,8cyclictest37-21ksoftirqd/321:55:013
3299632995513,10cyclictest131rcu_preempt23:01:333
3299632995510,6cyclictest131rcu_preempt21:13:533
329962999558,10cyclictest30-21ksoftirqd/222:52:382
329962999555,9cyclictest30-21ksoftirqd/221:32:222
329962999555,9cyclictest30-21ksoftirqd/221:32:212
3299629995516,10cyclictest30-21ksoftirqd/200:25:442
3299629995512,9cyclictest30-21ksoftirqd/223:10:022
3299629995511,7cyclictest30-21ksoftirqd/200:36:532
329962599558,10cyclictest23-21ksoftirqd/100:07:111
329962599557,6cyclictest23-21ksoftirqd/123:26:051
329962599555,8cyclictest23-21ksoftirqd/123:20:001
3299625995514,10cyclictest23-21ksoftirqd/122:55:021
3299622995533,16cyclictest275-21dbus-daemon20:45:020
3299622995533,16cyclictest275-21dbus-daemon20:45:010
3299622995528,12cyclictest3370284-21ssh23:41:140
3299622995527,13cyclictest3372575-21ssh23:46:450
3299622995527,13cyclictest3372575-21ssh23:46:450
3299622995526,14cyclictest3313756-21ssh21:28:180
3299622995526,13cyclictest3330881-21ssh22:08:390
3299622995525,14cyclictest3358849-21ssh23:14:360
329963299542,8cyclictest37-21ksoftirqd/323:55:013
3299632995418,12cyclictest37-21ksoftirqd/322:55:003
3299632995412,5cyclictest131rcu_preempt22:20:013
3299632995411,6cyclictest131rcu_preempt22:47:453
3299632995410,5cyclictest131rcu_preempt21:41:273
3299632995410,5cyclictest131rcu_preempt21:41:263
329962999548,9cyclictest30-21ksoftirqd/222:32:502
329962999547,9cyclictest30-21ksoftirqd/223:24:012
329962999546,5cyclictest30-21ksoftirqd/222:05:022
3299629995438,9cyclictest30-21ksoftirqd/221:25:222
3299629995413,12cyclictest30-21ksoftirqd/221:21:052
3299629995411,8cyclictest30-21ksoftirqd/200:25:032
3299629995411,8cyclictest30-21ksoftirqd/200:25:022
329962599547,6cyclictest23-21ksoftirqd/119:58:461
329962599545,9cyclictest23-21ksoftirqd/121:36:081
329962599543,8cyclictest23-21ksoftirqd/121:52:261
3299625995411,4cyclictest131rcu_preempt23:35:501
3299622995435,17cyclictest0-21swapper/022:22:040
3299622995427,12cyclictest3377009-21ssh23:57:090
329962299541,14cyclictest3312448-21latency_hist21:25:030
329963299539,4cyclictest131rcu_preempt21:46:323
329963299538,6cyclictest131rcu_preempt23:31:113
3299632995336,12cyclictest1-21systemd23:00:003
329962999538,10cyclictest30-21ksoftirqd/219:10:012
329962999537,8cyclictest30-21ksoftirqd/223:58:432
329962999536,9cyclictest30-21ksoftirqd/222:39:122
329962999534,8cyclictest30-21ksoftirqd/223:49:462
329962999534,8cyclictest30-21ksoftirqd/223:49:452
329962999534,15cyclictest3335340-21ssh22:19:042
3299629995333,9cyclictest30-21ksoftirqd/220:05:012
329962999532,17cyclictest3367883-21ssh23:35:302
3299629995313,10cyclictest30-21ksoftirqd/223:12:232
3299629995311,4cyclictest131rcu_preempt23:40:252
329962599537,10cyclictest23-21ksoftirqd/123:55:001
329962599535,7cyclictest23-21ksoftirqd/121:27:411
329962599535,12cyclictest23-21ksoftirqd/123:32:291
3299622995336,12cyclictest275-21dbus-daemon20:50:000
329962299533,3cyclictest131rcu_preempt21:30:320
329962299533,3cyclictest131rcu_preempt21:30:320
3299622995330,18cyclictest3361156-21sendmail-msp23:20:010
3299622995327,11cyclictest3347269-21ssh22:47:010
3299622995326,11cyclictest3376167-21ssh23:55:010
329962299532,18cyclictest3395050-21sendmail00:40:000
3299622995320,3cyclictest131rcu_preempt23:07:100
329963299527,5cyclictest131rcu_preempt23:17:373
329963299525,6cyclictest131rcu_preempt21:23:193
329963299524,7cyclictest37-21ksoftirqd/323:45:013
3299632995211,6cyclictest131rcu_preempt23:10:403
329962999529,10cyclictest30-21ksoftirqd/221:15:022
329962999527,9cyclictest30-21ksoftirqd/222:56:362
329962999527,8cyclictest30-21ksoftirqd/222:08:082
3299629995215,9cyclictest30-21ksoftirqd/223:17:282
3299629995212,7cyclictest0-21swapper/221:50:142
3299629995212,10cyclictest0-21swapper/219:20:012
3299629995212,10cyclictest0-21swapper/219:20:012
3299629995211,5cyclictest131rcu_preempt22:48:042
329962599526,9cyclictest23-21ksoftirqd/100:33:101
329962599526,7cyclictest23-21ksoftirqd/122:32:441
329962599526,6cyclictest23-21ksoftirqd/123:55:301
329962599524,9cyclictest23-21ksoftirqd/121:05:021
329962599524,13cyclictest23-21ksoftirqd/122:13:291
3299625995211,5cyclictest131rcu_preempt23:25:011
3299622995226,12cyclictest3386261-21ssh00:19:110
3299622995224,13cyclictest3309768-21ssh21:19:050
329963299519,5cyclictest131rcu_preempt22:31:563
329963299518,5cyclictest131rcu_preempt21:15:233
329963299518,12cyclictest37-21ksoftirqd/320:15:013
329963299517,7cyclictest37-21ksoftirqd/323:35:023
3299632995120,21cyclictest37-21ksoftirqd/322:05:033
3299632995111,5cyclictest131rcu_preempt19:49:463
329963299511,10cyclictest37-21ksoftirqd/319:20:013
329963299511,10cyclictest37-21ksoftirqd/319:20:003
3299632995110,6cyclictest131rcu_preempt22:43:113
329962999515,7cyclictest30-21ksoftirqd/223:50:582
329962999513,12cyclictest30-21ksoftirqd/221:55:572
3299629995113,9cyclictest30-21ksoftirqd/220:10:012
3299629995110,7cyclictest0-21swapper/220:17:012
3299629995110,6cyclictest131rcu_preempt22:30:012
329962599519,5cyclictest131rcu_preempt23:45:441
329962599519,5cyclictest131rcu_preempt23:45:431
329962599516,7cyclictest23-21ksoftirqd/122:37:311
329962599516,7cyclictest23-21ksoftirqd/121:11:431
329962599516,12cyclictest23-21ksoftirqd/122:42:181
329962599515,9cyclictest23-21ksoftirqd/100:17:101
329962599514,7cyclictest23-21ksoftirqd/123:41:531
329962599514,15cyclictest23-21ksoftirqd/121:55:071
3299625995110,4cyclictest131rcu_preempt23:05:191
3299622995123,4cyclictest131rcu_preempt21:41:590
3299622995123,4cyclictest131rcu_preempt21:41:580
329962299511,4cyclictest131rcu_preempt00:30:000
329963299507,6cyclictest131rcu_preempt00:15:303
329963299507,12cyclictest37-21ksoftirqd/321:27:393
329963299506,7cyclictest37-21ksoftirqd/323:08:563
329963299506,5cyclictest131rcu_preempt20:25:023
329963299505,9cyclictest37-21ksoftirqd/320:49:593
329963299505,9cyclictest37-21ksoftirqd/300:05:453
3299632995012,5cyclictest131rcu_preempt00:33:443
3299632995011,6cyclictest131rcu_preempt22:12:553
3299632995011,5cyclictest131rcu_preempt00:10:313
3299632995011,5cyclictest131rcu_preempt00:10:303
3299632995010,5cyclictest131rcu_preempt23:23:283
3299632995010,5cyclictest131rcu_preempt22:01:343
329962999507,8cyclictest30-21ksoftirqd/220:40:002
329962999502,9cyclictest30-21ksoftirqd/200:32:172
3299629995012,7cyclictest3319092-21rm21:40:352
3299629995012,7cyclictest3319092-21rm21:40:342
3299629995012,7cyclictest0-21swapper/222:43:372
329962599509,4cyclictest131rcu_preempt20:20:001
329962599509,4cyclictest131rcu_preempt19:55:011
329962599507,8cyclictest23-21ksoftirqd/122:27:421
329962599507,8cyclictest23-21ksoftirqd/121:30:371
329962599507,8cyclictest23-21ksoftirqd/121:30:371
329962599507,7cyclictest23-21ksoftirqd/121:00:021
329962599507,11cyclictest23-21ksoftirqd/123:10:421
329962599504,8cyclictest23-21ksoftirqd/120:10:001
3299625995016,11cyclictest23-21ksoftirqd/120:55:011
329962299505,39cyclictest0-21swapper/023:40:000
3299622995041,6cyclictest16950irq/224-800000022:54:590
3299622995024,10cyclictest3353512-21ssh23:01:350
329962299502,39cyclictest3367463-21modprobe23:35:000
3299622995010,10cyclictest12-21ksoftirqd/022:35:020
329963299499,5cyclictest131rcu_preempt23:45:293
329963299499,5cyclictest131rcu_preempt23:45:283
329963299498,7cyclictest131rcu_preempt20:35:023
329963299497,9cyclictest37-21ksoftirqd/300:21:143
329963299497,9cyclictest37-21ksoftirqd/300:21:133
3299632994928,15cyclictest275-21dbus-daemon20:00:013
329962999495,7cyclictest30-21ksoftirqd/219:30:012
329962999491,10cyclictest3305036-21latency_hist20:55:022
329962599496,6cyclictest23-21ksoftirqd/119:35:021
3299625994934,12cyclictest1-21systemd19:20:021
3299625994934,12cyclictest1-21systemd19:20:011
3299625994913,9cyclictest23-21ksoftirqd/119:10:011
3299622994917,10cyclictest12-21ksoftirqd/022:14:330
329962299491,42cyclictest3308118-21sort21:15:020
329963299487,10cyclictest37-21ksoftirqd/320:45:033
329963299487,10cyclictest37-21ksoftirqd/320:45:023
329963299484,8cyclictest37-21ksoftirqd/321:34:263
329963299484,8cyclictest37-21ksoftirqd/321:34:253
3299632994833,11cyclictest3305794-21cron21:10:003
3299632994820,16cyclictest37-21ksoftirqd/300:03:293
3299629994841,5cyclictest3363301-21kworker/2:000:05:462
3299629994812,6cyclictest0-21swapper/221:05:072
3299629994812,6cyclictest0-21swapper/220:25:172
3299629994812,6cyclictest0-21swapper/220:25:162
3299629994812,5cyclictest0-21swapper/221:01:212
3299629994811,7cyclictest0-21swapper/219:23:392
3299629994811,6cyclictest0-21swapper/219:45:092
3299629994811,5cyclictest0-21swapper/219:51:172
3299629994810,8cyclictest0-21swapper/220:45:112
329962599488,5cyclictest131rcu_preempt20:04:451
3299625994833,11cyclictest3467602-21systemd-journal19:45:001
329962299482,41cyclictest0-21swapper/022:30:020
329962299481,3cyclictest131rcu_preempt23:23:290
329963299476,9cyclictest37-21ksoftirqd/319:10:023
329963299474,9cyclictest37-21ksoftirqd/319:40:013
329963299473,7cyclictest37-21ksoftirqd/319:30:013
3299632994730,13cyclictest3467602-21systemd-journal19:10:013
329962999479,7cyclictest0-21swapper/219:55:292
329962999476,5cyclictest131rcu_preempt20:35:002
3299629994712,6cyclictest0-21swapper/220:14:492
3299629994711,7cyclictest0-21swapper/219:10:552
3299629994710,6cyclictest0-21swapper/219:40:542
329962599479,5cyclictest131rcu_preempt21:05:351
3299625994736,8cyclictest3300587-21sshd19:29:441
329962599472,8cyclictest23-21ksoftirqd/120:45:031
329962599472,8cyclictest23-21ksoftirqd/120:45:021
329962299474,3cyclictest131rcu_preempt00:23:110
329962299474,3cyclictest131rcu_preempt00:23:100
3299622994733,8cyclictest309-21in:imuxsock19:40:000
3299622994731,11cyclictest1-21systemd19:35:010
3299622994726,14cyclictest275-21dbus-daemon19:45:010
3299622994721,11cyclictest3364174-21ssh23:26:440
3299622994719,13cyclictest3318889-21cat21:40:020
329962299471,3cyclictest131rcu_preempt22:42:060
3299622994639,4cyclictest16950irq/224-800000020:10:010
3299622994623,4cyclictest0-21swapper/019:20:450
3299622994619,12cyclictest3325976-21ssh21:56:470
329962299461,3cyclictest131rcu_preempt00:14:110
329962299461,3cyclictest131rcu_preempt00:14:100
329963299453,9cyclictest37-21ksoftirqd/321:05:013
329962999458,8cyclictest0-21swapper/220:41:142
329962999458,8cyclictest0-21swapper/220:41:132
329962999458,8cyclictest0-21swapper/220:24:372
329962599451,5cyclictest0-21swapper/119:40:011
3299622994537,4cyclictest16950irq/224-800000019:20:020
3299622994537,4cyclictest16950irq/224-800000019:20:010
3299622994526,13cyclictest275-21dbus-daemon20:15:000
3299622994519,15cyclictest0-21swapper/020:00:000
329963299447,8cyclictest37-21ksoftirqd/321:00:023
329963299447,6cyclictest131rcu_preempt19:55:003
3299632994429,12cyclictest3302661-21sshd20:09:453
3299632994429,11cyclictest1-21systemd20:05:013
3299632994413,5cyclictest131rcu_preempt19:25:013
3299632994410,5cyclictest131rcu_preempt20:25:463
3299632994410,5cyclictest131rcu_preempt20:25:453
3299625994416,18cyclictest131rcu_preempt20:25:451
3299625994416,18cyclictest131rcu_preempt20:25:441
3299625994412,4cyclictest131rcu_preempt20:50:001
3299622994437,4cyclictest16950irq/224-800000019:30:010
3299622994437,4cyclictest16850irq/134-800000000:35:000
329963299439,5cyclictest131rcu_preempt20:40:003
329962599436,6cyclictest23-21ksoftirqd/120:40:021
329962599434,7cyclictest23-21ksoftirqd/119:25:011
329962599431,8cyclictest3303613-21sort20:25:021
3299622994329,10cyclictest309-21in:imuxsock20:35:020
329962299431,3cyclictest131rcu_preempt20:26:490
329962299431,3cyclictest131rcu_preempt20:26:490
329962599425,8cyclictest23-21ksoftirqd/119:15:001
329962599425,6cyclictest23-21ksoftirqd/120:35:011
329962299422,5cyclictest3301640-21modprobe19:50:000
329963299417,5cyclictest131rcu_preempt19:35:033
3299622994113,3cyclictest131rcu_preempt20:04:440
3299632994020,15cyclictest275-21dbus-daemon20:55:003
329962599402,8cyclictest23-21ksoftirqd/119:50:011
329962299403,24cyclictest131rcu_preempt20:59:570
3299622993927,7cyclictest309-21in:imuxsock20:55:010
3299622993830,6cyclictest12-21ksoftirqd/019:10:010
3299622993828,8cyclictest12-21ksoftirqd/021:05:000
3299622993721,13cyclictest275-21dbus-daemon19:55:010
3299622993623,10cyclictest12-21ksoftirqd/020:20:460
329962299352,6cyclictest1-21systemd20:17:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional