You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 20:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 16, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3179259991169,7cyclictest3196760-21ssh09:35:020
3179266999382,7cyclictest30-21ksoftirqd/211:25:022
3179259998783,2cyclictest0-21swapper/008:15:010
3179259998076,2cyclictest0-21swapper/010:45:030
317926699793,41cyclictest3193642-21ssh09:28:102
317926699742,37cyclictest3262573-21ssh12:02:072
317926699741,43cyclictest3253945-21ssh11:42:562
3179259997319,46cyclictest3261549-21sendmail12:00:010
3179259997319,46cyclictest3261549-21sendmail12:00:010
317926699707,45cyclictest3212426-21ssh10:10:032
317926699707,45cyclictest3212426-21ssh10:10:022
3179264997028,7cyclictest3184625-21sshd08:53:351
3179266996921,43cyclictest3250383-21cat11:35:022
317926699682,44cyclictest3188644-21ssh09:16:512
317926699681,63cyclictest3259132-21ssh11:54:492
3179264996849,10cyclictest23-21ksoftirqd/109:35:031
3179264996819,21cyclictest23-21ksoftirqd/109:30:031
317926699672,3cyclictest131rcu_preempt08:55:022
3179264996725,10cyclictest309-21in:imuxsock10:35:031
317926699662,40cyclictest3182868-21munin-run08:20:022
317926699662,31cyclictest3206837-21ssh09:57:432
317926699661,61cyclictest3260352-21ssh11:57:092
317926699661,61cyclictest3260352-21ssh11:57:092
317926699661,61cyclictest3236043-21ssh11:02:522
317926699661,60cyclictest3271734-21ssh12:24:052
317926699661,60cyclictest3271734-21ssh12:24:042
3179266996614,48cyclictest3183334-21sshd08:27:352
317926699661,32cyclictest3203683-21ssh09:50:182
317926699652,35cyclictest3184140-21munin-run08:45:002
3179266996520,41cyclictest30-21ksoftirqd/210:02:312
3179266996518,43cyclictest30-21ksoftirqd/210:45:232
317926699651,61cyclictest3220442-21ssh10:28:022
317926699651,60cyclictest3274038-21ssh12:29:222
317926699651,60cyclictest3269773-21ssh12:19:542
317926699651,60cyclictest3264699-21ssh12:06:582
317926699651,60cyclictest3264699-21ssh12:06:582
317926699651,60cyclictest3224418-21ssh10:36:492
317926699651,60cyclictest3215935-21ssh10:17:572
317926699651,60cyclictest3213761-21ssh10:13:122
317926699651,60cyclictest3198803-21ssh09:39:552
317926699651,60cyclictest3198803-21ssh09:39:542
317926699651,60cyclictest3194717-21ssh09:30:112
317926699651,60cyclictest3186133-21ssh09:11:062
317926699651,59cyclictest3256292-21ssh11:48:092
317926699651,59cyclictest3201428-21sh09:45:132
317926699641,60cyclictest3237129-21ssh11:05:052
317926699641,59cyclictest3277748-21ssh12:37:332
317926699641,59cyclictest3276124-21ssh12:34:072
317926699641,59cyclictest3252269-21ssh11:39:152
317926699641,59cyclictest3246463-21ssh11:26:022
317926699641,59cyclictest3242924-21ssh11:18:302
317926699641,59cyclictest3239381-21ssh11:10:112
317926699641,59cyclictest3230678-21ssh10:50:362
317926699641,59cyclictest3222097-21ssh10:31:342
317926699641,59cyclictest3222097-21ssh10:31:342
317926699641,59cyclictest3217366-21ssh10:20:502
317926699641,59cyclictest3199457-21ssh09:40:532
317926699631,59cyclictest3226943-21ssh10:42:342
317926699631,58cyclictest3266912-21ssh12:12:192
317926699631,58cyclictest3232824-21ssh10:55:302
317926699631,58cyclictest3190327-21ssh09:20:232
317925999633,8cyclictest3190112-21grep09:20:010
317926699621,57cyclictest3182042-21sshd08:04:352
317926699612,55cyclictest3179447-21munin-run07:15:012
317926699611,54cyclictest48-21kcompactd007:26:252
317926699592,52cyclictest364-21snmpd07:15:342
317926699591,54cyclictest3183120-21munin-run08:25:022
317926699591,54cyclictest3180461-21sshd07:30:352
317926699591,53cyclictest364-21snmpd08:38:332
3179264995928,9cyclictest23-21ksoftirqd/109:05:021
3179264995928,9cyclictest23-21ksoftirqd/109:05:011
317926699581,53cyclictest364-21snmpd07:47:142
317926699571,52cyclictest364-21snmpd08:32:432
317926699571,52cyclictest364-21snmpd08:32:422
3179266995646,6cyclictest3182340-21kworker/2:208:54:592
317926699563,48cyclictest364-21snmpd07:37:322
317926699562,50cyclictest3181745-21sshd07:55:342
317926699561,50cyclictest3185461-21munin-run09:10:012
317926699561,50cyclictest3184365-21sshd08:46:362
317926699561,50cyclictest3184365-21sshd08:46:352
3179259995612,39cyclictest12-21ksoftirqd/012:36:500
3179269995536,14cyclictest1-21systemd11:55:013
317926699552,49cyclictest3185141-21sshd09:00:352
317926699552,49cyclictest3185141-21sshd09:00:352
317926699551,49cyclictest364-21snmpd08:10:052
317926699551,49cyclictest364-21snmpd07:22:282
3179264995539,12cyclictest1-21systemd09:00:001
3179259995432,2cyclictest0-21swapper/010:30:020
3179259995427,21cyclictest3185435-21diskmemload10:15:010
3179259995427,12cyclictest3205435-21ssh09:54:480
3179259995427,11cyclictest3262269-21ssh12:01:250
3179259995417,8cyclictest3250301-21modprobe11:35:000
317925999541,5cyclictest0-21swapper/011:10:010
317925999534,32cyclictest16950irq/224-800000009:30:020
3179259995335,13cyclictest3467602-21systemd-journal10:40:000
3179259995332,7cyclictest12-21ksoftirqd/011:17:350
3179259995326,11cyclictest3235039-21ssh11:00:270
3179259995325,12cyclictest3273198-21ssh12:27:140
3179259995225,12cyclictest3241080-21ssh11:14:220
317925999522,42cyclictest3245816-21sh11:24:590
3179259995224,13cyclictest3211019-21ssh10:06:580
3179259995224,13cyclictest3211019-21ssh10:06:570
3179259995210,7cyclictest131rcu_preempt10:23:020
317926699511,46cyclictest364-21snmpd07:51:502
317926699511,46cyclictest364-21snmpd07:51:492
317925999516,9cyclictest12-21ksoftirqd/010:55:000
3179259995133,14cyclictest1-21systemd07:15:000
3179259995125,11cyclictest3266434-21ssh12:10:530
3179259995124,3cyclictest31915211ssh09:23:220
3179269995035,10cyclictest1-21systemd09:50:013
317926699502,43cyclictest364-21snmpd08:09:142
317926699502,43cyclictest364-21snmpd08:09:132
317926499504,34cyclictest221rcuc/112:05:011
3179264995010,23cyclictest0-21swapper/109:10:021
3179259995042,4cyclictest16850irq/134-800000011:45:000
3179259995024,5cyclictest19350irq/34-mmc110:40:380
3179259995024,11cyclictest3258753-21ssh11:53:430
3179269994935,11cyclictest37-21ksoftirqd/311:40:043
317926999493,3cyclictest131rcu_preempt07:30:013
3179269994932,12cyclictest3467602-21systemd-journal11:20:013
317925999495,5cyclictest131rcu_preempt12:20:030
317925999495,5cyclictest131rcu_preempt12:20:030
3179259994941,4cyclictest16850irq/134-800000011:30:010
3179259994940,5cyclictest16850irq/134-800000009:10:000
3179259994939,5cyclictest16950irq/224-800000010:00:000
3179259994935,8cyclictest3252515-21cron11:40:010
3179259994932,14cyclictest3209572-21ssh10:03:540
3179259994929,14cyclictest3467602-21systemd-journal07:45:010
3179259994928,16cyclictest3262128-21kworker/u8:312:08:270
3179259994928,16cyclictest3262128-21kworker/u8:312:08:270
317926499489,5cyclictest131rcu_preempt09:41:501
3179264994835,9cyclictest3257858-21sshd11:51:351
3179259994844,2cyclictest0-21swapper/008:05:010
317925999483,38cyclictest3182830-21latency_hist08:20:000
3179259994814,8cyclictest12-21ksoftirqd/010:34:570
3179259994814,8cyclictest12-21ksoftirqd/010:34:560
3179259994814,30cyclictest12-21ksoftirqd/009:11:200
3179269994729,12cyclictest3467602-21systemd-journal10:55:003
317926699472,41cyclictest0-21swapper/207:42:332
317926499474,3cyclictest131rcu_preempt12:28:271
3179264994731,12cyclictest3183829-21sshd08:35:351
3179259994726,17cyclictest31980231ssh09:37:530
3179259994726,17cyclictest31980231ssh09:37:530
317925999472,14cyclictest3467602-21systemd-journal12:33:060
3179259994721,23cyclictest3203220-21ssh09:49:460
3179259994717,14cyclictest3201131-21ssh09:45:000
3179259994711,32cyclictest12-21ksoftirqd/010:20:000
317926999465,8cyclictest37-21ksoftirqd/310:20:013
317926999462,3cyclictest131rcu_preempt11:29:223
3179259994632,10cyclictest1-21systemd08:40:010
3179259994632,10cyclictest1-21systemd08:00:010
3179264994529,12cyclictest1-21systemd10:35:011
3179264994529,12cyclictest1-21systemd10:35:001
3179264994528,14cyclictest3197468-21sshd09:36:351
3179264994528,14cyclictest3197468-21sshd09:36:341
3179264994520,17cyclictest23-21ksoftirqd/111:43:581
3179259994533,8cyclictest309-21in:imuxsock08:50:020
3179259994527,14cyclictest3269273-21sshd12:18:350
3179269994428,12cyclictest37-21ksoftirqd/308:00:003
3179269994424,15cyclictest275-21dbus-daemon09:20:003
317926999442,3cyclictest131rcu_preempt10:38:583
3179264994421,3cyclictest131rcu_preempt12:15:031
317926499441,12cyclictest3276658-21latency_hist12:35:021
3179259994437,4cyclictest16850irq/134-800000011:48:200
3179259994428,11cyclictest12-21ksoftirqd/008:10:030
3179259994428,11cyclictest12-21ksoftirqd/008:10:020
3179259994421,15cyclictest275-21dbus-daemon08:45:000
3179259994416,13cyclictest3233105-21ssh10:56:090
3179269994334,5cyclictest3234668-21latency_hist11:00:003
317926999431,4cyclictest131rcu_preempt10:07:023
317926999431,4cyclictest131rcu_preempt10:07:013
317926999431,3cyclictest131rcu_preempt12:36:133
3179264994326,12cyclictest275-21dbus-daemon07:55:011
3179264994326,12cyclictest275-21dbus-daemon07:55:001
3179264994325,14cyclictest3217085-21sh10:20:091
3179264994325,13cyclictest1-21systemd08:05:011
3179264994322,16cyclictest3216761-21cron10:20:001
3179259994332,6cyclictest3183352-21modprobe08:30:010
3179259994324,15cyclictest3180460-21sshd07:30:110
317926999422,9cyclictest3198057-21sh09:37:593
317926999422,9cyclictest3198057-21sh09:37:593
3179269994225,13cyclictest275-21dbus-daemon08:50:013
3179269994225,13cyclictest275-21dbus-daemon08:50:003
3179269994225,12cyclictest275-21dbus-daemon09:25:023
317926999422,3cyclictest131rcu_preempt10:47:123
317926999421,4cyclictest131rcu_preempt09:43:303
317926999421,3cyclictest131rcu_preempt09:10:583
317926699422,37cyclictest3179285-21latency_hist07:10:012
3179264994230,9cyclictest3183327-21sshd08:26:341
3179264994229,6cyclictest3180773-21sendmail-msp07:40:001
3179264994226,11cyclictest275-21dbus-daemon09:20:001
3179259994230,6cyclictest3179709-21sendmail07:20:010
3179259994227,11cyclictest3183641-21sed08:35:020
3179259994227,11cyclictest3183641-21sed08:35:010
3179269994133,4cyclictest3181273-21sh07:50:003
3179269994131,6cyclictest3250296-21sh11:34:593
3179269994129,9cyclictest37-21ksoftirqd/308:25:013
3179269994126,10cyclictest37-21ksoftirqd/310:27:433
3179269994124,13cyclictest0-21swapper/310:35:013
3179269994124,13cyclictest0-21swapper/310:35:013
317926999412,22cyclictest0-21swapper/310:20:043
317926999411,4cyclictest131rcu_preempt09:29:273
317926999411,3cyclictest131rcu_preempt12:32:513
3179269994111,6cyclictest131rcu_preempt09:35:013
3179264994124,12cyclictest275-21dbus-daemon10:05:011
3179264994121,8cyclictest3203449-21sort09:50:011
317926499411,3cyclictest131rcu_preempt07:50:011
3179259994131,6cyclictest3183126-21latency_hist08:25:010
3179259994120,15cyclictest275-21dbus-daemon09:00:010
3179269994027,9cyclictest3252568-21sh11:40:013
3179269994025,11cyclictest3179696-21cron07:20:003
317926999402,3cyclictest131rcu_preempt12:19:193
317926999402,3cyclictest131rcu_preempt11:11:433
317926999401,3cyclictest131rcu_preempt10:04:393
3179264994030,8cyclictest23-21ksoftirqd/111:25:281
3179264994026,11cyclictest23-21ksoftirqd/110:26:491
3179264994026,11cyclictest23-21ksoftirqd/109:52:271
3179264994025,11cyclictest23-21ksoftirqd/107:20:011
317926499401,4cyclictest131rcu_preempt08:00:021
3179259994033,4cyclictest19350irq/34-mmc107:25:010
3179259994032,5cyclictest19350irq/34-mmc108:46:120
3179259994032,5cyclictest19350irq/34-mmc108:46:120
317925999402,6cyclictest3180756-21munin-run07:40:000
317926999393,4cyclictest131rcu_preempt10:42:073
317926999393,4cyclictest131rcu_preempt10:10:533
3179269993928,9cyclictest37-21ksoftirqd/311:22:113
3179269993926,10cyclictest3184119-21sshd08:43:363
3179269993924,10cyclictest309-21in:imuxsock09:00:363
3179269993924,10cyclictest309-21in:imuxsock09:00:363
317926999391,3cyclictest131rcu_preempt11:46:073
317926499399,5cyclictest131rcu_preempt11:14:591
3179264993927,9cyclictest23-21ksoftirqd/111:06:211
3179264993926,10cyclictest23-21ksoftirqd/109:22:081
3179264993925,10cyclictest3467602-21systemd-journal12:40:011
317926499392,4cyclictest131rcu_preempt12:07:501
317926499392,4cyclictest131rcu_preempt12:07:501
3179264993924,11cyclictest23-21ksoftirqd/111:55:501
3179264993924,11cyclictest23-21ksoftirqd/111:55:491
3179264993918,16cyclictest275-21dbus-daemon08:20:011
317926499391,20cyclictest3237004-21ssh11:05:021
3179269993829,5cyclictest3261532-21munin-plugin-st12:00:003
3179269993829,5cyclictest3261532-21munin-plugin-st12:00:003
3179269993825,9cyclictest37-21ksoftirqd/309:51:533
3179269993825,11cyclictest37-21ksoftirqd/311:01:043
317926999382,4cyclictest131rcu_preempt12:22:563
317926999382,4cyclictest131rcu_preempt12:22:553
3179269993824,10cyclictest3467602-21systemd-journal08:20:013
3179269993822,10cyclictest3467602-21systemd-journal08:40:013
3179269993821,12cyclictest311-21rs:main3
317926999381,3cyclictest131rcu_preempt12:15:003
317926499384,3cyclictest131rcu_preempt08:50:021
317926499384,3cyclictest131rcu_preempt08:50:011
3179264993829,6cyclictest3183086-21sshd08:23:361
3179264993828,7cyclictest3243774-21sed11:20:031
3179264993826,9cyclictest23-21ksoftirqd/108:10:321
3179264993825,9cyclictest3270024-21sort12:20:021
3179264993825,11cyclictest23-21ksoftirqd/110:45:031
3179264993825,10cyclictest23-21ksoftirqd/111:39:311
3179264993825,10cyclictest23-21ksoftirqd/111:21:291
3179264993825,10cyclictest23-21ksoftirqd/110:51:511
3179264993820,13cyclictest275-21dbus-daemon12:25:011
3179264993820,13cyclictest275-21dbus-daemon12:25:011
317926999372,7cyclictest131rcu_preempt09:09:593
3179269993726,8cyclictest37-21ksoftirqd/312:01:173
3179269993725,10cyclictest37-21ksoftirqd/312:10:023
3179269993725,10cyclictest37-21ksoftirqd/312:10:013
3179269993724,9cyclictest37-21ksoftirqd/308:28:403
3179269993713,19cyclictest275-21dbus-daemon11:10:013
317926499374,29cyclictest3182786-21kworker/u8:008:30:031
317926499374,29cyclictest3182786-21kworker/u8:008:30:031
3179264993730,4cyclictest3255466-21sh11:46:121
3179264993726,9cyclictest23-21ksoftirqd/110:58:181
3179264993721,11cyclictest23-21ksoftirqd/107:43:041
3179259993724,9cyclictest12-21ksoftirqd/009:05:010
3179259993724,9cyclictest12-21ksoftirqd/009:05:000
3179259993712,5cyclictest131rcu_preempt07:45:350
3179269993627,6cyclictest309-21in:imuxsock07:15:003
3179269993616,14cyclictest275-21dbus-daemon08:05:003
317926999361,3cyclictest131rcu_preempt12:27:593
3179264993627,6cyclictest3187687-21latency_hist09:15:021
3179264993625,8cyclictest23-21ksoftirqd/108:45:021
3179264993623,10cyclictest23-21ksoftirqd/108:10:021
3179264993623,10cyclictest23-21ksoftirqd/108:10:011
317926499361,6cyclictest150-21pr/ttyS207:34:591
317926499361,4cyclictest131rcu_preempt10:50:021
3179269993521,10cyclictest37-21ksoftirqd/309:00:003
3179264993526,7cyclictest23-21ksoftirqd/110:00:001
3179264993524,8cyclictest23-21ksoftirqd/111:31:331
3179264993523,10cyclictest23-21ksoftirqd/110:06:011
3179264993523,10cyclictest23-21ksoftirqd/110:06:001
3179259993522,9cyclictest3181494-21sshd07:50:360
3179259993522,9cyclictest3181494-21sshd07:50:350
317925999351,7cyclictest3173374-21kworker/u8:007:30:020
317926999342,3cyclictest131rcu_preempt07:25:003
3179269993423,9cyclictest37-21ksoftirqd/307:40:013
3179269993423,8cyclictest37-21ksoftirqd/308:10:013
3179269993423,8cyclictest37-21ksoftirqd/308:10:003
3179269993420,11cyclictest37-21ksoftirqd/308:55:023
3179269993417,12cyclictest275-21dbus-daemon07:35:013
317926999341,3cyclictest131rcu_preempt07:10:023
3179264993418,12cyclictest275-21dbus-daemon07:15:011
3179269993322,8cyclictest37-21ksoftirqd/308:10:023
3179269993317,13cyclictest275-21dbus-daemon07:45:003
3179264993325,6cyclictest23-21ksoftirqd/107:26:361
3179264993325,5cyclictest3212803-21sh10:10:501
3179264993325,5cyclictest274-21cron07:24:591
3179269993124,4cyclictest3183621-21sh08:35:013
3179269993124,4cyclictest3183621-21sh08:35:003
3179269993122,6cyclictest3181586-21latency_hist07:55:023
3179269993122,6cyclictest3181586-21latency_hist07:55:013
317926499314,3cyclictest131rcu_preempt07:10:021
3179259993119,8cyclictest12-21ksoftirqd/007:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional